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A program  evaluating  the  application  of  data  capture  techniques  to  the  entry  of  shipping 
data  at  MAC  air  cargo  terminals  is  in  progress.  As  a part  nf  this  program  six  airmen 
from  the  truck  dock  of  the  MAC  air  cargo  terminal  at  Dover  A FH  p ueipated  at 
MITHK  in  tests  comparing  three  alternative  modes  of  data  entry:  an  on-line  keypad/ 
display,  a digital  recorder,  and  manual  checksheets.  Data  items  wore  entered  from 
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I’p'i'1'  i >l‘dt.  The  contract  is  s|>onsored  by  the  Fleetronie  Systems 
Division,  Air  Force  Systems  Command,  llanseom  Air  Force  Base, 
Massachusetts. 

The  initial  testing  <>f  data  entry  ti  ehniques  rejmrted  here  was  carried 
cut  with  the  help  id  six  airmen  from  the  truck  dock  of  the  MAC  air  cargo 
terminal  at  Dover  Air  Force  Ha.se:  Sgt.  .lames  Nolan,  Kddie  Garcia, 

Kerry  Kennedy,  Sewell  Umghrun,  dim  ltathbun,  and  .Jeffrey  Tinsley.  These 
men  worked  effectively  as  test  operators  in  the  evaluation  of  alternative 
data  entry  modes.  Their  participation  was  arranged  with  the  cooperation 
of  Colonel  duel  .1.  Neeley,  FSAF,  Commander  of  the  -130  Aerial  Port 
Squadron  at  Dover. 

Several  jieople  at  MITHK  contributed  significantly  to  this  test  program. 
Warren  K.  Anderson  designed  and  implemented  the  computer  software 
prov  iding  sequence  control  for  the  on-line  keypad  'display  terminal,  and 
served  as  observer  for  test  sessions  involving  that  device.  He  was  also 
res|»>nsible  for  software  mediating  computer  storage  and  processing  ofall 
shipment  data,  print  out  of  advance  shipment  data  in  checksheet  form,  and 
computer  processing  of  test  records.  .Joanne  H.  Czuladn  developed  the 
software  for  computer  storage  and  preliminary  screening  of  records  pro- 
duced by  the  digital  recorder.  She  also  served  as  observer  for  test  sessions 
using  the  checksheet  mode  of  data  entry.  Mary  .lane  Ashmore  assisted  in 
analysis  of  transaction  tin  e records. 
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SECTION  I 


INTRODUCTION 


The  Military  Airlift  Command  (MAC)  is  responsible  for  air 
movement  of  DoD  cargo  throughout  the  world.  This  cargo  consists  of 
thousands  of  different  shipments  and  separate  pieces.  Cargo 
selected  for  air  movement  is  essential  military  material  requiring 
records  to  be  kept  concerning  its  disposition  while  moving  through 
the  air  transport  system.  Thus  the  handling  of  air  cargo  is 
accompanied  oy  a heavy  burden  of  data  handling  as  well.  It  is  vital 
that  this  data  processing  be  made  as  efficient  as  possible,  so  that 
record  keeping  will  not  slow  cargo  movement.  In  the  MAC  Integrated 
Management  System  (MACIMS)  program,  various  ways  are  being 
considered  to  improve  air  cargo  data  processing. 

Preliminary  analysis  indicates  that  data  processing  at  MAC's  air 
cargo  terminals  can  be  improved  by  facilitating  data  entry  at  the 
truck  docks.  Here  at  the  truck  dock,  cargo  destined  for  export 
first  comes  into  the  air  transport  system.  A3  each  piece  of  cargo 
is  unloaded,  its  shipping  label  mu3t  be  examined  to  determine  proper 
disposition,  and  pertinent  data  recorded  in  some  way  for  subsequent 
processing.  Several  alternative  modes  have  been  considered  for 
improving  data  entry  at  the  truck  dock,  as  discussed  in  the  first 
volume  of  tnis  report  seriestlJ. 

One  potential  mode  of  data  entry  would  involve  image  capture  of 
shipping  label  data  at  the  truck  dock,  either  by  conventional 
pnotography  or  by  video  recording  techniques.  Such  an  approach 
offers  potential  advantages  in  truck  dock  use.  A complete  3et  of 
data  about  each  piece  of  cargo  could  be  recorded  in  the  few  seconds 
it  might  take  to  photograph  its  shipping  label.  To  explore  the 
feasibility  of  photographing  shipping  label  data  in  the  truck  dock 
environment,  several  MITRE  personnel  tested  that  technique  earlier 
this  year,  at  MAC's  mechanized  air  cargo  terminal  at  Dover  Air  Force 
Base,  Delaware.  The  results  of  that  visit  were  summarized  in  the 
second  volume  of  tnis  report  series^]. 

An  obvious  disadvantage  of  image  capture  techniques,  of  course, 
is  that  imaged  data  would  have  to  be  processed  in  some  way  and 
transcribed  into  digital  form,  before  becoming  available  for 
reference  within  the  air  transport  system.  Other  modes  of  data 
entry  at  the  truck  dock  may  prove  more  promising.  It  might  be 
possible,  for  example,  to  provide  a member  of  an  unloading  crew  with 
some  kind  of  keyboard/display,  which  he  could  use  to  input  data  over 
a direct,  on-line  connection  to  a computer.  Alternatively,  he  might 
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record  shipping  data  in  some  way,  perhaps  by  keying  items  into  a 
digital  recorder  or  simply  writing  them  on  a checksheet,  with 
subsequent  data  input  to  the  computer  delayed  until  truck  unloading 
has  been  completed. 

The  final  evaluation  of  any  data  entry  mode  can  only  be  made  by 
measuring  its  effectiveness  in  full-scale,  day-to-day  use  at  the 
truck  dock.  But  such  full-scale  evaluation  can  be  expensive,  and  it 
is  not  practical  to  test  all  possible  data  entry  modes  in  that  way. 
Some  preliminary  screening  of  alternative  modes  must  be  done  first, 
in  successive  stages  of  analysis,  initial  laboratory  testing,  and 
follow-on  field  testing,  so  that  only  those  modes  which  prove  likely 
candidates  will  be  considered  for  full-scale  operational  evaluation. 

Some  of  tne  more  promising  modes  of  data  entry  were  identified 
on  the  basis  of  preliminary  analysis It  was  decided 
to  conduct  a program  of  initial  testing  at  MITRE  to  compare  three 
methods  of  accomplishing  the  data  entry  job.  The  three  alternatives 
chosen  for  initial  testing  were  those  suggested  above  - U3e  of  an 
on-line  keypad/display,  a digital  recorder,  or  manual  checksheets. 
The  present  report  documents  the  results  of  this  initial  test 
program. 

A detailed  description  of  the  test  design  and  procedures  used  is 
presented  in  Section  II  of  this  report.  Testing  was  conducted  using 
facilities  in  MITRE '3  Data  Handling  Applications  Center  (abbreviated 
DHAC  in  the  remainder  of  this  report).  Performance  wa3  measured  for 
the  several  candidate  input  modes,  as  the  test  operators  entered 
data  from  photographed  shipping  labels  simulating  truck  loads  of 
cargo.  The  simulated  loads  used  for  testing  varied  in  shipment-to- 
piece  ratio  and  in  the  availability  df  advance  shipment  data,  in 
order  to  permit  estimation  of  the  effects  of  these  two  load  factors 
on  data  entry  performance. 

Performance  was  measured  in  terms  of  both  speed  and  accuracy. 
Section  III  of  this  report  presents  an  analysis  of  the  time  required 
for  data  entry,  as  influenced  by  tha  input  mode  used,  by  composition 
of  the  test  loads,  and  by  session  to  session  changes  in  performance 
as  test  operators  became  more  familiar  with  their  tasks.  This 
analysis  of  data  entry  speed  is  reported  in  terms  of  overall  time 
required  to  complete  a test  load,  the  tiirae  required  to  complete 
characteristic  sequences  of  data  entry,)  and  the  time  required  for 
individual  data  input  transactions.  Also  included  in  Section  III  is 
an  analysis  of  keying  activity  for  the  two  data  entry  modes 
requiring  keyboard  inputs. 


Accura Cj  of  data  entry  is  discussed  in  Section  IV  of  this 
report.  Error  rates  are  analyzed  to  determine  the  effect  of  data 
entry  mode,  lead  factors,  and  individual  differences  among  the  test 
operators.  Errors  are  categorized  fcy  type,  including  a distinction 
between  errors  of  data  content  and  errors  of  data  format.  In 
addition  to  analysis  of  entry  errors  committed  by  the  test 
operators,  Section  IV  also  presents  an  analysis  of  operator 
performance  in  detecting  and  correcting  errors  during  review  of  pre- 
stored advance  shipment  data. 

Altnough  performance  measures  can  provide  an  objective  basi3  for 
the  comparative  evaluation  of  different  data  entry  modes,  it  was 
also  considered  important  to  record  the  opinions  of  the  test 
operators  themselves.  These  men,  with  firsthand  experience  in  the 
actual  unloading  job  at  the  MAC  truck  docks,  we  e asked  to  evaluate 
the  different  data  entry  modes  they  had  used  in  t.ie  laboratory  in 
terms  of  tneir  potential  application  in  the  real  job.  The  results 
of  that  operator  evaluation  are  presented  in  Section  V of  thi3 
report . 

Much  valuable  information  was  obtained  in  this  initial  test 
program,  and  its  overall  results  should  provide  a useful  guide  in 
developing  improved  techniques  for  MAC  air  cargo  data  entry.  Some 
follow-on  laboratory  testing  will  be  required,  however,  along  with  a 
subsequent  program  of  on-the-job  field  testing,  before  a full-scale 
system  can  be  recommended  for  operational  use.  Section  VI  of  this 
report  presents  recommendations  for  such  a follow-on  test  program. 

Certain  specific  aspects  of  the  initial  test  program  merit  more 
detailed  discussion  than  could  conveniently  be  included  in  the  body 
of  tnis  report.  Several  special  topics  are  discussed  at  greater 
length  in  a series  of  Appendices.  Appendix  A describes  the 
capabilities  and  technical  characteristics  of  the  two  data  entry 
devices  tested  here  - the  on-line  terminal  and  the  digital  recorder. 
Appendix  B describes  the  operational  logic  of  the  interactive 
software  providing  sequence  control  for  the  on-line  terminal. 
Appendix  C discusses  certain  deficiencies  noted  in  the  MITFE- 
designed  keyboard  for  the  digital  recorder  and  proposes  an  improved 
design  for  future  use.  Appjndix  D discusses  the  detailed  format  of 
the  checksheets  used  for  testing  and  includes  a sample  set  of 
checksheets.  Appendix  E provides  an  extended  analysis  of  format 
errors  in  use  of  the  digital  recorder.  Appendix  F describes  the 
questionnaire  formats  used  in  operator  evaluation  of  the  different 
data  entry  modes  tested,  and  provides  a sample  set  of  those 
questionnaires . 
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SECTION  II 


TEST  DESIGN 


During  the  period  12-22  May,  1975,  six  men  from  the  436  Aerial 
Port  Squadron,  Dover  Air  Force  Base,  participated  a3  test  operators 
in  an  initial  evaluation  of  alternative  data  entry  techniques  for 
potential  future  use  at  the  truck  docks  of  MAC  air  cargo  terminals. 
Tnis  test  pregram  was  conducted  at  the  Data  Handling  Applications 
Center,  The  MITRE  Corporation,  Bedford,  Massachusetts. 

Te3t  operators  workec  to  record  data  from  facsimile  shipping 
labels  photographed  at  Dover  AFB  from  truck-delivered  cargo  in  early 
April.  That  is  to  say,  the  operators  did  not  actually  have  to 
handle  real  packages,  but  simply  entered  the  data  from  a pack  of 
photographed  labels,  plus  tag  identifier  and  disposition  code  for 
each  "piece”.  Simulated  truck  loads  used  during  testing  each 
consisted  of  the  labels  from  48  pieces  of  cargo. 

Different  test  loads  were  used  to  examine  the  effect  of 
different  load  factors.  Half  of  the  loads  represented  a high 
shipment-to-piece  ratio  (36  shipments,  S/P=.75)  and  half  a low  ratio 
(12  shipments,  S/P=.25).  Half  of  the  te3t  loads  simulated  a high 
level  (.75)  of  advance  shipment  information  available,  and  half  an 
unusually  low  level  (.25). 

Three  modes  of  data  entry  were  tested:  1)  an  on-line  terminal, 

a handheld  keypad/display  device  which  the  operators  used  to  enter 
data  in  direct  interaction  with  a computer,  with  computer  control  of 
tne  input  sequence  and  computer  prompting  for  correction  of  detected 
input  errors;  2)  a digital  recorder,  involving  key  entry  to  create 
magnetic  tape  records  of  digital  data  which  were  subsequently  input 
to  a computer  for  processing  and  storage;  3)  a checksheet . or 
rather  a pack  of  checksheets,  on  which  the  operators  wrote  down  the 
necessary  data  for  each  test  load. 

Each  operator  worked  for  a day  with  each  of  the  three  data  entry 
modes,  in  four  sessions  each  day  using  different  test  loads, 
providing  an  overall  total  of  72  test  sessions.  Performance 
measures  consisted  of  the  time  required  to  complete  each  session, 
the  time  required  for  a variety  of  specific  data  entry  transactions 
within  each  session,  a count  of  data  entry  errors  of  different 
kinds,  and  a measure  of  the  operators'  success  in  detecting  and 
correcting  errors  in  the  pre-stored  data  for  each  test  load. 
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At  the  end  of  the  daily  test  sessions,  each  operator  was  asked 
to  write  an  evaluation  of  >-he  particular  data  entry  mode  he  had  ju3t 
used,  recording  his  judgments  on  a question  sheet  formatted  for  that 
purpose.  At  the  end  of  tnree  days  of  testing,  each  operator  was 
asked  to  evaluate  all  three  data  entry  modes  in  comparison  with  one 
another. 

The  data  entry  task,  the  modes  of  data  entry,  and  the  procedures 
used  for  testing,  are  described  in  greater  detail  in  the  remainder 
or  this  section. 


THE  DATA  ENTRY  TASK 

In  testing  data  entry  modes  there  must  be  some  data  to  be 
entered.  Photographed  shipping  labels  were  used  to  provide  such 
data,  grouped  into  simulated  test  loads  wh.ch  varied  in  shipment-to- 
piece  ratio  and  in  the  availability  of  advance  shipment  data. 

Errors  and  omissions  were  included  in  the  advance  data  to  te3t  error 
detection  performance. 

Shipping  Labels 

To  create  the  corpus  of  shipment  data  needed  for  testing,  MITRE 
personnel  spent  two  days  at  MAC's  mechanized  air  cargo  terminal  at 
Dover  Air  Force  Base,  photographing  several  hundred  shipping  labels 
from  pieces  of  truck-delivered  cargo  destined  for  air  transport 
overseas.  The  results  of  that  photographic  expedition  have  been 
described  in  volume  two  of  this  report  series (2],  The 
photographic  negatives  were  enlarged  to  produce  prints  of 
approximately  true  size,  thus  providing  fair  facsimile  of  the  labels 
themselves,  one  of  wnich  is  illustrated  in  Figure  1. 

The  standard  military  shipment  label,  DD  Form  1387,  contains  1 2 
fields  of  data.  Fir3t  is  the  transportation  control  number  (TCN),  a 
group  of  17  symbols  which  uniquely  identifies  each  shipment.  The 
TCN  i3  followed  oy  further  data  about  the  shipment,  including 
required  delivery  date  (RDD),  an  optional  project  code,  priority, 
consignor  (the  FROM  field),  aerial  ports  of  embarkation  (POE)  and 
debarkation  (POD),  and  tne  consignee.  The  bottom  line  indicates  the 
total  pieces  of  cargo  in  the  shipment  and  the  number,  weight  and 
cube  of  tne  specific  piece  bearing  this  label. 

For  shipments  comprised  of  more  than  one  piece  of  cargo,  the  TCN 
and  the  general  shipment  data  would  be  the  same  from  one  label  to 
the  next.  Only  the  piece-specific  data  would  differ.  For  a multi- 
piece shipment,  it  should  be  possible  to  shorten  the  data  entry  task 
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Figure  1.  Photographed  Shipping  Label  with  Tag  Identlfie 
and  Disposition  Code  Added 


somewhat,  by  entering  a full  set  of  data  for  the  firjt  piece 
unloaded,  but  then  simply  entering  a shipment  identifier  and  the 
piece-specific  data  for  each  subsequent  piece  from  that  shipment. 
Such  shortcut  procedures  were  in  fact  used  in  this  test  program. 

In  actual  air  terminal  operations,  r.ot  all  data  items  on  the 
shipping  label  would  be  considered  equally  important.  Probably  the 
TCN,  priority  and  POD  are  the  most  important  items  for  efficient 
cargo  handling,  whereas  the  other  items  of  shipment  data  are  used 
for  ir're  general  record  keeping  purposes.  In  this  initial  test 
pregram,  however,  no  attempt  was  made  to  draw  such  distinctions.  It 
was  assumed  that  ail  label  data  had  to  be  entered.  Timing  measures 
were  taken  separately  for  different  portions  of  the  data  entry  task, 
as  will  be  noted  later,  so  that  the  test  results  can  be  interpolated 
to  predict  performance  for  a real  task  involving  entry  of  only 
partial  data. 

Actually,  tne  operators  in  this  test  program  were  required  to 
enter  two  extra  data  items  for  each  piece  of  cargo,  in  addition  to 
the  shipping  label  data.  At  the  lower  right  of  each  photographed 
shipping  label  was  attached  a special  adhesive  tag  displaying  a 
computer-printed  tag  number,  uniquely  identifying  the  piece,  and  a 
disposition  code,  both  of  which  were  included  in  the  data  entry 
tas*.  This  special  tag  can  be  seen  in  Figure  1.  In  this 
illustration  the  disposition  code  (DISP)  is  a 5-symbol  group 
identifying  a cart  in  which  the  piece  of  cargo  i3  assumed  to  have 
been  placed. 

In  actual  cargo  handling,  the  operator  at  the  truck  dock  would 
have  to  decide  the  appropriate  disposition  for  each  piece.  In  this 
laboratory  test  situation,  the  label  for  every  piece  had  already 
been  assigned  a disposition.  That  is  to  say,  the  test  operators  did 
not  have  to  handle  real  cargo  or  make  decisions  about  it.  They 
simply  entered  data  about  each  piece  of  simulated  cargo  based  on  the 
facsimile  label  and  tag  information. 

The  following  list  indicates  in  more  detail  just  what  items 
comprised  the  data  entry  task. 

TCN  1HA/N  (alphanumeric)  symbols.  The  last  3 symbols  of 

the  real  TCN's  were  omitted  since  they  are  always  XXX 
except  when  changed  to  denote  partial  and  split 
shipments . 

INDEX  2N,  a simple  alternative  to  the  TCN,  designating  a 

shipment  from  the  load  list  of  advance  shipment  data. 
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Use  of  an  index  to  shortcut  TCN  entry  is  discussed 
later  in  this  report. 

3N,  generally  the  Julian  notation  for  required 
delivery  date,  although  occasionally  some  other  code 
group  appears.  Operators  were  instructed  not  to  try 
to  convert  conventional  calendar  Jate3  which  appeared 
on  some  labels,  but  simply  to  omit  them. 

3A/N,  a code  sometimes  included  for  accounting 
purposes,  but  often  omitted. 

IN,  either  1,  2,  3,  or  9. 

6A/N  consignor  code.  Operators  were  not  asked  to 
enter  names  and  addresses  of  suppliers,  but  simply  the 
6-symbol  agency  code  when  it  appeared. 

3A  code  designating  port  of  embarkation,  in  this  case 
always  DOV  3ince  the  labels  used  were  all  photographed 
at  Dover. 

3A  code  designating  port  of  debarkation,  the  overseas 
destination  for  each  piece  of  cargo.  For  some  labels 
which  happened  to  omit  the  POD,  the  appropriate  code 
was  printed  onto  the  photograph  so  that  the  operator 
did  not  have  to  figure  it  out. 

6A/N  code  designating  the  receiving  agency.  No  names 
or  addresses  were  entered  here,  but  only  the  6-symbol 
code  if  it  appeared. 

IN,  designating  the  piece  number.  In  actual  transport 
operations  this  number  might  occasionally  have  two 
digits  or  even  more  for  larger  shipments. 

IN,  designating  the  total  number  of  pieces  in  the 
shipment.  In  this  test  program,  the  largest  shipments 
simulated  contained  8 pieces. 

IN  to  AN,  designating  the  piece  weight  in  pounds. 

IN  to  3N,  designating  piece  size  in  cubic  feet. 

6N,  representing  what  is  thought  to  be  an  adequate 
length  code  to  identify  each  piece  uniquely  during  its 
movement  within  the  air  transport  system.  Tag  codes 
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were  assigned  in  sequence  to  each  piece  in  a test 
load,  simulating  the  order  of  unloading  at  a truck 
dock,  but  with  one  break  included  at  an  arbitrary 
point  in  that  sequence  for  each  load. 

DISP  1A  or  3A  or  5A/N,  representing  codes  for  the  assumed 

disposition  of  each  piece  of  cargo.  Pieces  for  local 
distribution  (L)  or  requiring  special  handling  (S) 
were  assigned  single-letter  codes.  Pieces  heavy 
enough  to  require  direct  forklift  to  the  pre-pallet 
storage  areas  were  coded  by  their  3-letter  POD. 

Smaller  pieces  were  assigned  5-symbol  ''odes  simulating 
the  numbers  of  carts  in  which  they  were  assumed  to 
have  been  placed.  These  disposition  code3  were 
invented  for  purposes  of  testing;  some  other  code 
scheme  might  well  prove  better  in  actual  cargo 
handling  operations. 


During  testing,  all  of  these  data  items  did  not  have  to  be 
entered  for  every  piece.  Where  data  items  were  missing  from  the 
label  they  were  perforce  omitted  in  data  entry.  For  a shipment 
where  advance  data  were  made  available,  those  data  items  did  not 
have  to  be  entered  unless  found  to  be  wrong  in  comparison  with  the 
label.  The  minimum  data  entry  required  would  be  for  a label 
representing  a single-piece  shipment  with  correct  advance  data:  in 

that  case,  shipment  weight  and  cube  would  match  the  piece  weight  and 
cube,  so  the  only  data  entry  required  would  be  the  TCN  (or  INDEX) 
shipment  designator  followed  by  the  tag  ard  dispostion  codes  for  the 
piece.  For  multi-piece  shipments  with  correct  advance  data,  the 
task  required  entry  of  piece-specific  data  as  well,  i.e.,  piece 
number,  weight  and  cube. 

Test  Loads 

To  preoare  the  data  base  needed  to  support  the  test  program,  the 
photographed  shipping  labels  were  grouped  into  12  stacks  with 
different1  labels  in  each.  These  stacks  simulated  the  truck  loads  of 
arriving  cargo  whose  data  would  be  entered  during  12  test  sessions. 
That  is  to  say,  as  a test  operator  moved  from  one  data  entry  mode  to 
another,  (‘rem  one  session  to  the  next,  he  found  a new,  different 
"lo  0ata  waiting  to  be  entered.  If  he  had  to  enter  the  same 

load  data  session  after  session,  he  might  come  to  have  that  set  of 
data  memorized  by  the  end  of  the  test  series.  Certainly  it  would 
come  to  seem  more  and  more  familiar. 
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But  if  different  test  loads  are  to  be  used,  by  different 
operators  using  different  means  of  data  entry,  than  it  is  important 
that  the  test  loads  either  be  equivalent  in  respect  to  task 
difficulty,  or  that  they  differ  in  factors  which  are  measured  and 
taken  into  account.  In  the  design  of  this  test  program  both 
approaches  were  employed. 

The  12  test  loads  were  equivalent  in  siue,  each  containing  the 
labels  from  48  pieces  of  cargo.  It  wa3  estimated  that  the  data  from 
48  labels  would  take  something  less  than  an  hour  to  enter,  which 
seemed  about  right  for  test  purposes.  The  ordering  of  labels  within 
each  stack  of  48  was  completely  random.  For  a multi-piece  shipment, 
the  several  labels  for  the  different  pieces  could  be  mixed  anywhere 
in  the  stack.  Just  as  if  a van  load  of  cargo  had  gotten  jumbled  up 
en  route  to  the  truck  dock  and  had  to  be  unleaded  in  random  order. 

In  actual  operations,  of  course,  vans  are  sometimes  packed  more 
neatly,  with  some  effort  made  to  keep  all  pieces  of  each  shipment 
reasonably  close  to^ciner.  Thus  the  test  situation  was  designed  to 
represent  a worst  case  for  data  entry. 

The  test  loads  were  also  made  equivalent  in  terms  of  the  image 
quality  of  their  photographed  labels.  Image  quality  cf  the 
photographs  taken  at  Dover  was  generally  good,  but  varied  from 
excellent  to  poor  for  particular  labels  12 J.  Labels  were 
assigned  to  different  test  leads  in  such  a way  that  each  load 
included  37-39  labels  of  excellent  or  good  legibility,  and  9-11 
labels  of  fair  or  poor  quality,  i.e.,  labels  which  were  blurred 
carbon  copies  or  perhaps  smudged  with  dirt  and  hard  to  read. 

The  12  test  lc.is  differed  in  two  factors  controlled  in  the 
experimental  design.  Loads  wore  constructed  so  as  to  represent  two 
levels  of  shipment-to-piece  ratio,  and  all  loads  were  presented 
during  the  test  'eries  under  two  conditions  simulating  high  ana  low 
levels  of  availability  of  advance  shipment  data. 

Shipment-to-Plece  Ratio 

Presumably,  it  should  be  easier  to  enter  the  data  for  one 
shipment  of  four  pieces  than  for  four  shipments  of  one  piece  each. 
For  the  multi-piece  shipment,  the  complete  set  of  general  shipment 
data  would  only  have  to  be  entered  once,  after  which  only  a subset 
of  piece-specific  data  would  be  entered  for  each  additional  piece  in 
that  shipment.  Thus  an  important  factor  influencing  the  difficulty 
of  the  data  entry  task  should  prove  to  be  the  shipment-to-piece 
ratio  in  each  load.  The  more  multi-piece  shipments  in  a load,  then 
the  lower  the  S/P  ratio,  the  more  data  items  on  the  shipp* ng  label 
can  be  omitted  on  the  average,  and  the  easier  the  data  entry  task 
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During  the  photographic  expedition  to  Dover,  the  observed  S/P 
ratio  for  truck-delivered  cargo  was  approximately  .50,  representing 
an  average  of  about  two  pieces  per  shipment.  The  distribution  of 
shipment  size  was  quite  skewed,  with  many  shipments  having  only  a 
single  piece  and  a few  shipments  hiving  multiple  pieces(2]. 

It  was  decided  to  construct  test  loads  at  both  a higher  and 
a lower  S/P  ratio,  bracketing  the  observed  mean,  to  measure  the 
effect  of  this  load  factor  on  data  entry  performance. 

Six  of  the  12  test  loads  were  constructed  to  represent  an  S/P 
ratio  of  .75,  where  the  Uij  labelled  pieces  came  from  36  different 
shipments.  The  otner  six  test  loads  were  constructed  with  an  S/P  of 
.25,  where  the  1*0  pieces  comprised  only  12  different  shipments.  The 
number  of  snipments  of  different  sizes  in  these  "high"  and  "low" 
te3t  loads  is  shown  in  Table  2-1.  Some  of  the  shipments  cf  large1- 
size  were  created  artificially,  by  replicating  a photographed  label 
several  times  and  marking  different  piece  numbers  on  each  copy,  to 
construct  test  loads  with  the  desired  S/P  ratios. 

Table  2-1 

Variation  in  Shipment  Size  for  Test  Loads 
with  High  and  Low  S/P  Ratios 


Shipment  Size  Number  of  Shipments  in  Load 


Once  the  12  test  load3  of  different  shipment-to-piece  ratio  had 
been  constructed,  tne  snipping  label,  tag  and  disposition  data  for 
the  US  pieces  in  each  load  were  keypunched  and  stored  on  disk  in  the 
LriAC  computer.  Thus  a complete  set  of  data  for  each  load  was 
available  for  comparison  with  data  entered  subsequently  by  the  tes*’ 
operators,  to  determine  how  accurately  they  performed  the  data  entry 
task  under  different  test  conditions. 

For  each  test  load,  the  computer  was  programmed  to  print  out  two 
3et3  of  checksheets,  one  set  displaying  the  pre-stored  shipping 
label  data  for  one-quarter  of  the  shipments  (and  one-quarter  of  the 
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pieces)  in  the  load,  and  the  other  set  displaying  pre-stored  data 
for  the  remaining  three-quarters.  These  checksheets  were  given  to 
the  operators  during  testing  to  provide  either  a low  level  (.25)  of 
advance  shipment  data  or  a high  level  (.75),  respectively.  The 
partitioning  of  shipments  used  to  simulate  these  low  and  high  levels 
of  advance  shipment  data  is  indicated  in  Table  2-2  for  test  loads  of 
high  and  low  S/P  ratio. 


Table  2-2 


Availability  of  Advance  Shipment  Data 


Number  of  Shipments  in  Load 
with  Advance  Data  Available 


S/LMLfr  1-7.5). 


S/JLL9W.L..SSI 


Shipment 
(Number  of 
1 
2 

3 

4 
6 
8 


Size  Advance  Data 
Pieces)  Low  (.25)  High  (.75) 
7 23 

1 1 

1 1 

2 


Advance  Data 

Low  1.25)  High  (.75) 

1 1 

2 

1 1 

2 

2 

1 1 


The  format  used  for  the  advance  data  printout  is  illustrated  by 
the  sample  set  of  checksheets  presented  in  Appendix  D.  Advance  data 
for  each  shipment  in  the  load  were  displayed  on  a separate  page, 
with  the  items  arranged  in  a format  corresponding  to  the  layout  of  a 
shipping  label  to  facilitate  coraparision  during  review  of  advance 
data  by  the  test  operators.  It  should  be  noted  that  tag  and 
disposition  were  pot  included  in  the  advance  data,  since  these  would 
be  determined  during  the  unloading  of  actual  cargo,  and  that  piece- 
specific  data  were  Included  only  for  single-piece  shipments  where 
such  data  are  known  in  advance. 

The  advance  data  printed  out  for  each  load  contained  some 
errors.  Some  of  those  had  been  introduced  deliberately  in  order  to 
te3t  the  ability  of  test  operators  to  detect  and  correct  errors  in 
their  review  of  the  advance  data.  It  turned  out  that  still  other 
errors  were  introduced  accidentally  during  the  keypunching  of 
shipping  label  data  for  computer  storage.  In  general,  errors  in  the 
advance  data  consisted  of  two  types,  missing  data  items  and  wrong 
items.  The  subsequent  analysis  of  test  operator  performance 
compared  error  detection  for  both  types  of  error. 
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The  two  levels  of  advance  shipment  data  used  in  this  initial 
test  program  simulated  situations  which  might  actually  be 
encountered  in  truck  dock  unloading  operations.  A high  level  of 
advance  data  represents  the  usual  case.  Indeed  the  proportion  ol 
shipments  with  advance  data  will  often  exceed  the  .75  level  tested 
here.  A low  level  of  advance  data  would  represent  a truck  load  of 
cargo  which  arrives  with  inadequate  advance  documentation.  It  wa3 
expected  that  this  factor,  the  availability  of  advance  data,  would 
have  a significant  effect  on  data  entry  performance,  and  that 
expectation  was  confirmed  in  testing. 


MODES  OF  DATA  ENTRY 

Three  modes  of  data  entry  were  compared  in  this  initial  test 
program:  a portable  on-line  terminal  consisting  of  a handheld 

keypad  with  visual  display;  a digital  recorder,  permitting  data 
entry  to  be  performed  as  an  off-line  task;  and  -he  manual  printing 
of  data  entries  on  specially  formatted  checksheets.  The 
characteristics  of  these  three  input  modes,  as  used  for  the  data 
entry  task,  are  described  in  the  following  paragraphs. 

On-Line  Terminal 

As  used  here,  the  word  "terminal"  refers  to  a device  connected 
to  a computer,  permitting  direct,  on-line  interaction  during 
performance  of  the  data  entry  task.  The  word  is  not  intended  to 
refer  to  any  larger  facility  such  as  an  air  cargo  terminal.  A 
photograph  of  a test  operator  working  with  the  on-line  terminal  is 
shown  in  Figure  2.  The  on-line  terminal  was  connected  to  the 
computer  presently  being  used  in  MITRE  s Data  Handling  Applications 
Center,  a NOVA  600  minicomputer  manufactured  by  Data  General 
Corporation. 

A close  view  of  the  keypad/display  for  the  on-line  terminal  is 
shown  in  Figure  3.  This  device  is  manufactured  by  Termiflex 
Corporation,  Nashua,  New  Hampshire,  and  is  often  called  simply  "the 
termiflex".  The  terminal  itself  weighs  only  1.5  pounds.  Its 
display  can  present  two  lines  of  10  symbols  each.  Its  keyboard  has 
an  alphanumeric  cluster  similar  to  that  which  has  become  standard  on 
Touch-Tone ® telephones , plus  some  extra  control  keys  below  and  to 
the  left.  A more  detailed  description  of  this  device  is  provided  in 
Appendix  A to  this  report. 

The  terminal  is  designed  to  be  held  in  the  left  hand,  with  the 
right  hand  U3ed  for  keying  data  entries.  Numeric  data  can  be  keyed 
in  a straightforward  manner.  Alphabetic  entries  require  depression 
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Using  Handheld  On-Line  Terminal 


Figure  Close  View  of  Handheld  Kevpad/Display 


of  one  of  three  shift  keys  on  the  side  of  the  termiflex,  by  fingers 
of  the  left  hand,  to  designate  which  of  three  letters  is  intended 
when  the  right  hand  presses  a key  on  the  face  of  the  device.  This 
requirement  for  simultaneously  combined  action  from  fingers  of  the 
two  hands  seems  awkward  on  first  use  of  the  device,  particularly  in 
the  key  entry  of  mixed  alphanumeric  data. 

Key  entries  appear  on  the  bottom  line  of  the  associated  display, 
at  the  position  denoted  by  a cursor  symbol,  which  is  not  visible  in 
Figure  3.  When  the  bottom  display  line  is  full,  i.e.,  vrhen  10 
symbols  have  been  entered,  it  moves  upward  displacing  the  old  top 
line  from  view.  Further  data  entry  is  displayed  in  the  new  bottom 
line.  In  use  of  the  termiflex  for  the  task  of  entering  cargo  data, 
the  data  items  were  entered  one  at  a time,  each  in  a single  display 
line  except  for  the  truncated,  14-symbol  TCN  which  required  two 
lines  of  display. 

Since  this  device  wa3  used  on-line,  the  computer  could  be 
programmed  to  interpret  any  selected  key  entry  as  calling  some 
particular  function  needed  in  sequence  control.  It  can  be  seen  in 
Figure  3 that  six  special  control  keys  were  assigned  such  programmed 
functions: 


• YES/ENTER  to  signal  the  end  of  a data  entry,  or  to  answer 

affirmatively  a computer-posed  question. 

• NO/CANCEL  to  cancel  a garbled  data  entry  currently  being 

input,  or  to  respond  negatively  to  a question. 


• BACKSPACE  (labelled*—)  to  cancel  the  last  symbol  in  a 

current  data  entry. 

• BACKUP  to  cancel  the  current  entry  and  return  to  the 

last  previous  step  in  the  data  entry  sequence. 


• RESTART/STOP  to  interrupt  the  data  entry  sequence  to  enter  a 

revised  TCN,  or  to  stop  the  sequence. 

• NEW  DELAY  to  divert  from  the  data  entry  sequence 

temporarily  to  define  a new  delay  parameter  for 
transient  advisory  displays,  as  discussed  below. 


Since  the  termiflex  wa3  used  on-line,  the  computer  could  control 
the  contents  of  the  display.  This  capability  was  exploited  in 
several  ways  when  programming  sequence  control  for  the  data  entry 
task.  A3  a simple  example,  when  the  operator  keyed  BACKSPACE,  the 
computer  regenerated  the  entire  two-line  display  minus  its  last 
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symbol.  Tne  resultant  visual  effect  was  to  see  the  display  blink  as 
the  cursor  apparently  moved  backward  one  space  to  erase  the  la3t 
symbol. 

During  the  data  entry  sequence  the  computer  wrote  "prompts"  to 
the  termiflex  display.  Figure  3 shows  the  prompt  for  entry  of  a 
TCN.  When  entry  of  a data  item  was  completed  using  the  ENTER  key, 
the  computer  checked  the  input  Item  against  stored  criteria  for 
validating  data.  If  an  entry  seemed  correct,  with  proper  number  of 
symbols,  type  of  symbols,  etc.,  then  the  computer  displayed  the 
appropriate  prompt  for  the  next  item  to  be  entered.  Thu3  the  data 
entry  task  proceeded  as  a sequence  of  discrete  transactions. 

If  the  computer  detected  an  error  in  tne  format  of  a data  entry, 
it  was  programmed  to  "beep"  an  auditory  signal  inside  the  termiflex, 
to  display  a transient  error  message  for  a set  interval  of  time,  ? id 
then  to  display  a prompt  for  the  re-entry  of  that  data  item.  In 
certain  instances  two  transient  error  messages  would  appear  in 
succession,  to  provide  guidance  in  alternative  means  of  error 
correction.  The  time  interval  chosen  for  the  display  of  such 
messages  was  set  under  computer  control,  and  could  be  changed 
arbitrarily  by  using  the  NEW  DELAY  key  to  initiate  a short  on-line 
transaction  to  enter  whatever  interval  is  desired.  During  testi  lg, 
an  interval  of  1.5  seconds  was  used  during  each  operator's  first 
session  with  the  on-line  terminal.  This  interval  was  shortened  to 
.5  seconds  for  subsequent  sessions. 

If  tne  operator  himself  detected  an  error  during  data  entry,  he 
could  use  the  BACKSPACE  key  to  correct  it.  If  he  became  confused, 
he  could  use  the  CANCEL  key  to  abort  the  transaction.  The  computer 
simply  regenerated  the  prompt  for  the  data  item,  without  any 
intervening  error  messages.  If  he  wanted  to  change  some  previous 
data  entry,  he  could  use  the  BACKUP  key,  which  caused  the  computer 
to  step  backward  through  the  data  entry  sequence  one  item  at  a time. 
If  the  operator  decided  at  any  point  that  he  had  to  revise  his  entry 
for  a TCN,  he  could  use  the  RESTART  key  to  abort  the  current  data 
entry  sequence  and  start  over  again.  Such  options  permit 
flexibility  in  the  use  of  an  on-line  terminal.  They  may  seldom  be 
used  but  are  helpful  when  needed. 

Aside  from  these  implicit  options,  always  available  to  him,  the 
operator  could  be  offered  an  explicit  choice  by  the  computer.  At 
the  end  of  data  entry  for  each  piece  of  cargo,  he  was  asked  if  there 
were  data  for  another  piece  to  be  entered.  He  would  use  the  YES  key 
to  request  continuation  of  the  data  entry  sequence,  and  finally  the 
NO  key  to  signal  the  end  of  a load.  For  shipments  with  advance  data 
available,  the  control  sequence  required  the  operator  to  review  the 
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data  Item  by  item,  to  confirm  correct  items  and  change  incorrect 
ones,  for  the  first  piece  "unloaded"  for  that  shipment.  For  a new 
TCN,  he  simply  entered  the  necessary  shipment  data  for  the  first 

piece.  For  subsequent  pieces,  in  both  cases,  the  operator  was  given 

a choice  as  to  whether  he  wished  to  review  the  stored  shipment  data, 

and  he  used  the  YES  or  NO  keys  to  indicate  his  decision. 

Other  niceties  of  program  control  of  the  interactive  data  entry 
sequence  included  computer  anticipation  of  highly  probable  inputs. 

In  entry  of  port  of  embarkation,  for  example,  the  computer  prompt 
read  POE=DOV?  The  operator  could  approve  this  "guess"  with  a single 
keystroke,  rather  than  having  to  key  the  three  letter  code.  For 
piece  identification  the  computer  predicted  the  next  tag  to  be 
entered  by  adding  1 to  the  last,  assuming  that  pieces  are  tagged  in 
their  order  of  "unloading".  That  is  to  say,  if  the  last  piece 
entered  had  the  tag  325687,  the  tag  prompt  for  the  next  piece  would 
read  TAG=325688?  The  operator  could  confirm  this  prediction  with  a 
single  keystroke  unless  he  had  just  reached  that  point  in  the  te3t 
load  where  a break  in  the  tag  sequence  had  been  inserted. 

The  control  program  mediating  on-line  data  entry  (and  also 
performance  recording,  as  discussed  later)  was  completely  checked 
out  in  3hakedown  runs  prior  to  testing.  No  program  flaws  appeared 
during  actual  testing.  A more  detailed  outline  of  the  sequence 
control  logic  used  for  the  on-line  terminal  is  presented  in  Appendix 
B to  this  report,  along  with  suggestions  for  improvement  based  on 
problems  observed  during  testing. 

It  should  be  noted  that  for  each  test  load  the  operator  of  the 
on-line  terminal  was  given  a set  of  checksheets  displaying  the 
advance  data  available  for  that  load,  as  described  later  in  this 
report.  He  did  not  need  to  refer  to  these  checksheets,  however, 
because  the  computer  displayed  advance  data  item  by  item  for  review 
at  hi3  terminal.  He  could  simplify  his  data  entry  task  somewhat  by 
referring  to  an  index  which  accompanied  the  checksheets,  listing  by 
TCN  all  shipments  with  advance  data.  If  a piece  of  cargo  came  from 
a shipment  on  that  list,  the  operator  could  simply  enter  the 
shipment's  two-digit  index  number  instead  of  the  full  TCN.  Each 
"new"  shipment  in  the  load  was  automatically  assigned  an  index  by 
the  computer.  If  the  operator  bothered  to  note  this  new  index,  by 
writing  its  TCN  on  the  index  list,  he  could  use  this  index  when 
entering  data  for  subsequent  pieces  of  that  shipment. 

Digital  Recorder 

By  "digital  recorder"  is  meant  a device  which  permits  key  entry 
of  data  off-line  to  create  a digital  record,  perhaps  on  magnetic 
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tape,  which  can  subsequently  be  played  back  and  transmitted  for 
computer  storage.  (A  keypunch  would  serve  the  same  function,  but 
keypunch  equipment  is  more  cumbersome  and  more  expensive.)  The 
digital  recorder  used  in  this  initial  test  program  is  called  the 
Source  2001  portable  data  terminal  by  its  manufacturer , MSI  Data 
Corporation,  Costa  Mesa,  California.  A photograph  of  the  digital 
recorder  being  used  by  a test  operator  is  shown  in  Figure  4.  A more 
detailed  description  of  the  capabilities  of  this  device  is  presented 
in  Appendix  A to  this  report. 

Previous  research  at  MITRE  had  provided  some  measures  of  data 
entry  time  and  errors  using  the  digital  recorder,  and  had  indicated 
tiiat  this  device  might  prove  suitable  for  air  cargo  data  entry 
(Sutherland,  1974).  It  was  decided  to  undertake  further  testing  of 
the  digital  recorder  under  conditions  wh.ch  simulate  more 
realistically  the  truck  dock  data  entry  task. 

A MITRE-designed  keyboard  previously  used  with  the  digital 
recorder  was  refurbished  for  use  in  the  present  study.  A close  view 
of  this  special  keyboard  is  shown  in  Figure  0.  Tne  keyboard  was 
held  In  one  hand  by  a pistol  grip  fastened  to  its  bottom  (not 
visible  in  the  photo),  and  key  entries  were  made  with  the  other 
hand.  Different  groups  of  keys  were  coded  by  color.  The  alphabetic 
keys  on  the  left  were  blue.  The  numeric  keys  on  the  right  were 
green.  The  special  line  indicator  keys,  above  and  to  the  left  of 
the  numeric  cluster,  were  red. 

In  practice,  this  keyboard  layout  did  not  prove  optimum  for  the 
data  entry  task.  The  problem  is  one  of  format  control.  In  a data 
entry  sequence  using  the  on-line  terminal,  the  computer  can  prompt 
the  entry  of  each  data  item  and  sc  "knows”  what  the  item  is.  The 
operator  of  a digital  recorder  does  not  enjoy  this  advantage.  As  he 
enters  a sequence  of  data  items  he  must  strike  extra  keys  to 
indicate  which  item  will  be  entered  next.  The  most  straightforward 
keyboard  design  would  provide  one  specially-labelled  indicator  key 
for  each  type  of  data  item  to  be  entered.  The  keyboard  actually 
used  did  not  have  enough  extra  keys  available  for  that  purpose. 

The  expedient  used  in  this  initial  test  program  was  to 
categorize  tne  different  data  items  on  a shipping  label  into  several 
"lines",  and  then  use  the  spare  keys  on  the  keyboard  to  indicate 
which  line  of  data  was  to  be  entered  next.  To  mark  the  boundaries 
of  the  several  data  fields  within  each  line,  it  was  necessary  that 
the  operators  key  an  ErJTER  button  after  each  data  item.  This  line 
entry  logic  was  explained  to  each  operator  during  hi3  first  test 
session  using  the  digital  recorder,  with  the  aid  of  the  special 
instructional  display  reproduced  in  Figure  6.  That  display  remained 
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available  to  serve  as  a ready  reference  throughout  all  sessions 
using  the  digital  recorder,  and  can  be  discerned  in  Figure  ;; . 

Just  as  there  was  no  on-line  sequence  control  provided  by  a 
computer  to  the  user  of  the  digital  recorder,  neither  was  there  any 
on-line  feedback  from  computer  checking  of  data  entry  errors.  The 
operator  himself  had  to  stay  alert  to  notice  errors  as  he  made  them. 
The  keyboard  was  configured  to  provide  similar  means  :>f  error 
correction  to  those  used  with  the  on-line  terminal.  The  HACKCPACF 
key  permitted  correction,  i.e.,  erasure  and  re-entry,  of  particular 
characters.  A CAHCL  key  permitted  erasure  of  the  data  field 
currently  being  entered.  A backup  capability  of  sorts  was  provided 
by  using  double-keying  of  a line  indicator  to  begin  re-entry  of  all 
data  items  in  a line,  or  to  correct  a TCN.  Operators  were  reminded 
of  these  error  correction  procedures  by  a supplementary  reference 
display,  reproduced  In  Figure  7. 

Aside  from  producing  a record  of  key  inputs  cn  magnetic  tape, 
this  particular  digital  recorder  also  generates  a printed  record  on 
a strip  of  paper.  At  any  time,  the  symbols  most  recently  entered 
can  be  viewed  (with  some  difficulty)  through  a small  display  window 
in  the  front  of  the  recorder  device.  Te3t  operators  using  the 
digital  recorder  seldom  looked  at  this  paper  display,  usually  being 
content  simply  to  re-enter  data  if  they  became  confused  and 
doubtful.  On  several  occasions  during  the  test  sessions  the  3trip 
printer  was  not  operating  properly,  which  may  have  contributed  to 
the  operators'  reluctance  to  rely  on  its  display. 

It  should  be  noted  that  for  each  te3t  load  the  operator  of  the 
digital  recorder  was  given  a set  of  checksheets  displaying  r.he 
advance  data  available  for  that  load,  as  described  later  in  this 
report.  He  could  refer  to  these  sheets  to  determine  whether 
corrections  to  advance  data  needed  to  be  recorded.  He  did  net  net- 
to  record  data  items  which  were  already  correct  in  the  advance 
shipment  information.  Thus  by  reviewing  checksheet  data  the 
operator  could  shorten  hi3  data  entry  task.  He  could  also  refer  to 
the  index  accompanying  the  checksheets  to  designate  shipments  by  a 
short  index  number  rather  than  by  TCN,  just  as  he  could  when  using 
the  on-line  terminal.  For  a new  shipment,  he  could  assign  an  index 
number  himself  if  he  wished,  using  the  appropriate  keying  sequence 
noted  in  Figure  6. 

Anticipating  the  results  of  this  study,  which  are  discussed  in 
later  sections  of  this  report,  the  logic  governing  the  data  entry 
sequence  and  the  procedures  for  error  correction  both  seemed  to  pose 
problems  to  users  of  the  digital  recorder.  Evidence  to  support  that 
conclusion  was  derived  from  an  analysis  of  format  errors  made  using 
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this  device,  summarized  in  Appendix  E,  and  tend3  to  contradict  the 
operators'  expressed  evaluation  of  all  data  entry  modes  as  easy  to 
learn.  If  the  digital  recorder  is  to  be  considered  for  actual  use 
in  this  data  entry  job,  it  will  be  necessary  to  redesign  it3 
keyboard  and  consequently  the  logic  of  data  input.  Suggestions  for 
an  improved  keyboard  layout  are  presented  in  Appendix  C to  this 
r>  port. 

Manual  Check-sheets 

The  third  mode  of  data  entry  which  was  tested  did  not  require 
any  equipment  r.t  all,  except  for  pencil  and  paper.  The  paper 
consisted  of  a set  of  specially  formatted  checksheets  on  which  the 
operator  could  print  the  necessary  data  for  each  piece  of  cargo  in  a 
test  load.  Figure  S shows  a test  operator  working  witn  a set  of 
checksheets. 

The  checksheets  were  printed  out  by  the  NOVA  computer,  from  the 
stored  data  base  of  snipping  label  information,  in  advance  of  the 
test  sessions.  For  each  test  load,  two  sets  of  checksheets  were 
available,  one  providing  a low  level  of  advance  snipment  data  and 
the  other  a high  level.  The  test  operator  was  given  whichever  set 
was  appropriate,  depending  upon  his  particular  test  schedule.  A 
sample  set  of  checksheets  is  presented  in  Appendix  D to  this  report. 

The  checksheets  displayed  advance  data  one  shipment  to  a page, 
with  data  items  arranged  to  correspond  to  the  format  of  a shipping 
label  for  convenient  comparison.  Operators  were  instructed  to  write 
in  any  missing  data  items,  including  piece-specific  data  and 
possible  omissions  from  the  advance  shipment  data,  and  to  cross  out 
and  write  over  any  items  which  were  wrong  in  t.ie  advance  data.  For 
"new"  shipments  all  data  items  had  to  be  written  in,  and  blank 
checksheets  were  provided  for  that  purpose. 

Each  set  of  checksheets  began  with  a cover  sheet  which  listed 
the  TCN's  for  t.nose  shipments  with  advance  data,  along  with  a two- 
digit  index  number  associated  with  each  TON.  These  index  numbers 
were  displayed  prominently  on  the  checksheets  and  so  could  be  used 
by  the  operator  to  speed  his  search  through  the  pack  looking  for  the 
checksheet  for  a particular  shipment.  For  new  shipments,  the  te3t 
operators  were  encouraged  to  add  TCN's  to  the  index  list  if  they 
wished  to  do  30.  Often  they  did  index  new  shipments,  particularly 
multiple-piece  shipments  whose  checksheets  would  have  to  be  accessed 
several  times  during  a test  session. 

Aside  from  the  use  of  checksheets  for  data  entry,  sets  of 
checksheets  were  also  provided  as  supplementary  reference  material 
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to  test  operators  working  with  the  on-line  terminal  and  with  the 
digital  recorder,  as  noted  earlier  in  the  descriptions  of  those  two 
data  entry  modes.  The  operator  of  the  on-line  terminal  used  only 
the  index  list  attached  to  the  checksheets,  to  permit  shortcut  entry 
of  index  number  rather  than  TCN  to  designate  shipments  having 
advance  data.  The  operator  of  the  digital  recorder  U3ed  the  index 
list  in  the  same  way,  and  referred  to  the  checksheets  as  necessary 
to  review  the  correctness  of  advance  data.  In  neither  of  those  two 
modes  did  the  operator  have  to  write  data  onto  the  checksheets. 
Neither  of  those  modes  require  that  checksheets  of  advance  data  be 
u3ed,  but  the  checksheets  (or  at  least  a load  index  list)  may  prove 
a helpful  reference  when  available. 


TEST  PROCEDURES 

Having  defined  the  data  entry  task  to  be  performed,  and  the 
modes  of  data  entry  to  be  tested,  the  remaining  aspects  of  test 
design  have  to  do  with  the  selection  and  training  of  operators,  the 
ached  ling  and  conduct  of  test  sessions,  and  the  observation, 
measurement  and  evaluation  of  performance.  The  procedures  used  in 
this  initial  test  program  are  described  in  the  following  paragraphs. 

Test  Operators 

Six  men  from  the  1)36  Aerial  Port  Squadron,  Dover  Air  Force  Base, 
were  recrui  ad  to  participate  as  test  operators.  These  men  were 
drawn  from  the  unloading  crews  currently  working  the  truck  dock  at 
the  Dover  air  cargo  terminal.  As  a group  they  may  be  considered  a 
representative  sample  of  MAC  personnel  assigned  to  this  job, 
including  men  with  different  degrees  of  experience  in  cargo 
handling,  ranging  from  a few  months  to  several  years.  None  of  these 
men  had  any  particular  prior  experience  in  data  entry,  and  none  of 
them  now  perform  any  specialized  clerical  tasks  at  the  truck  dock. 
These  men  all  have  normal  vision  and  nc  manual  impairment. 

I 

This  initial  test  program  was  conducted  at  the  ESD/MITRE  Data 
Handling  Applications  Center  in  Bedford,  Massachusetts.  For 
convenience  m scheduling,  test  operators  were  invited  to  this 
laboratory  facility  in  groups  of  three,  each  group  working  there  for 
several  days,  the  first  group  on  13-il5  May  1975  and  a second  group 
on  20-22  May.  i 

Each  group  of  test  operators  was  welcomed  to  the  DHAC  with  a 
short  briefing  for  general  orientation.  They  were  told  something  of 
the  continuing  effort  to  upgrade  MAC  data  processing  and  the 
perceived  need  to  improve  data  entry  at  the  truck  docks.  They  were 
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given  a snort  summary  description  of  the  three  data  entry  modes  to 
be  tested.  They  were  told  that  the  purpose  of  this  initial  testing 
was  to  determine  whether  any  of  these  data  entry  modes  look  good 
enough  to  warrant  subsequent  field  testing.  They  were  told  that 
performance  measures  would  be  taken  of  the  speed  and  accuracy  of 
data  entry,  and  that  they  would  be  asked  to  make  their  own 
evaluation  of  each  data  entry  mode.  They  were  assured  that  it  was 
t a device  or  mode  which  was  being  tested  and  not  themselves,  and 
that  anonymity  would  be  preserved  in  any  subsequent  reporting  of 
individual  performance. 

Sample  photographs  of  shipping  labels  were  distributed  to  the 
test  operators  to  "emind  them  of  the  data  format,  and  the  various 
items  comprising  general  shipment  data  and  specific  piece  data  were 
reviewed.  The  concept  of  pre-stored  data  simulating  availability  of 
advance  shipment  information  was  discussed,  and  also  the  differences 
in  shipment-to-pi ece  ratio  which  can  occur  from  load  to  load.  It 
was  explained  that  test  loads  would  vary  in  both  of  these  factors, 
and  so  the  operators  should  not  be  surprised  if  the  data  entry  task 
proved  easier  for  one  load  than  another. 

A work  schedule  was  posted  showing  each  of  the  test  operators 
which  data  entry  mode  he  would  use  for  each  of  the  several  days  of 
testing.  Test  operators  were  then  introduced  to  the  three  MITRE 
observers  who  would  work  with  them  during  the  test  sessions,  and  the 
first  session  was  begun.  No  attempt  was  made  to  familiarize 
operators  with  a data  entry  mode  in  any  period  of  nands-on  use  prior 
to  testing.  All  training  was  conducted  on  the  Job,  so  that  any 
problems  encountered  could  be  observed  and  learning  measured  in 
terms  of  changes  in  Job  performance. 

Sched'il  i ng 

Each  operator  worked  four  sessions  using  a particular  data  entry 
mode  in  each  day  of  testing.  Each  session  involved  data  entry  for  a 
different  test  load,  with  either  a high  or  low  shipment-to-piece 
ratio,  combined  with  either  a high  or  low  availability  of  advance 
shipment  data.  The  daily  sequence  in  which  the  operators  used  the 
three  data  entry  modes  was  counterbalanced  so  that  potential  order 
effects  would  not  bias  the  subsequent  analysis  of  performance 
comparing  modes.  As  an  example,  if  there  were  a tendency  for 
operators  to  learn  general  aspects  of  the  job  and  work  better  from 
day  to  day,  giving  their  best  performance  with  whatever  mode  is  used 
last,  then  to  permit  a fair  comparison  of  different  data  entry  modes 
it  would  be  important  that  different  operators  use  them  in  different 
order.  Similarly,  the  sequence  in  which  the  operators  worked  ith 
different  types  of  test  loads,  from  session  to  session,  was 
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randomized  so  that  the  effects  of  learning  how  to  use  a data  entry 
mode  would  not  introduce  any  consistent  bias  in  the  subsequent 
analysis  of  load  factors.  The  resulting  schedule  of  test  sessions 
which  embodied  these  constraints  is  summarized  in  Table  2-3* 


Table  2-3 

Test  Scheduling  of  Data  Entry  Mode  and  Load  Factors 

Day:  1 2 3 

Session:  1231»123*»1234 

' „‘St 

Operator: 

C OT  DR  CS 

LL  HH  LH  HL  HH  HL  LL  LH  LL  LH  HH  HL 

B DR  CS  OT 

LH  LL  HL  HH  LH  HL  LL  HH  HL  LH  HH  LL 

A CS  OT  DR 

HL  HH  LH  LL  HH  LL  HL  LH  HL  HH  LH  LL 

E OT  CS  DR 

LH  HL  LL  HH  HL  LL  HH  LH  LH  LL  HH  HL 

D CS  DR  OT 

LH  HL  LL  HH  LL  LH  HL  HH  HH  LL  LH  HL 

F DR  OT  CS 

HL  HH  LH  LL  LL  HH  HL  LH  HH  LH  HL  LL 

Note:  For  each  te3t  operator,  designated  here  by  letters  A through 

F,  tne  first  row  indicates  the  data  entry  mode  used  on  each 
test  day:  OT  = on-line  terminal;  DR  = digital  recorder;  CS 

= checksheet.  The  second  row  indicates  characteristics  of 
the  test  load  used  in  each  session:  HL  = high  S/P  ratio 
and  a low  level  of  advance  shipment  data;  LH  = low  S/P  and 
high  advance  data;  etc. 


The  test  schedule  was  contrived  in  such  a way  that  on  any  day 
each  of  the  three  operators  in  a group  was  using  a different  mode  of 
data  entry.  This  was  necessary  because  only  one  on-line  terminal 
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and  one  digital  recorder  were  available  for  use  in  the  DHAC,  and 
because  the  computer  program  provided  control  for  only  one  on-line 
device  at  a time. 

Test  Environment 

No  attempt  was  made  to  simulate  the  truck  dock  environment  in 
this  initial  test  program.  As  noted  earlier,  the  test  operators  did 
not  have  to  handle  real  cargo,  nor  make  decisions  about  its 
disposition.  They  were  given  no  waybills  to  check.  They  simply 
entered  or  recorded  data  from  a stack  of  photographed  shipping 
labels. 

Test  operators  did  not  have  to  move  about,  from  one  piece  of 
cargo  to  another,  dodging  forklifts  and  other  workers,  nor  did  they 
have  to  carry  any  of  the  data  entry  gear  they  were  using.  Instead, 
they  worked  comfortably  seated  at  tables.  They  did  not  have  to 
strain  to  read  labels  in  the  dimly  lit  interior  of  a van.  In  the 
laboratory  a high  ambient  illumination  was  provided  by  overhead 
fluorescent  lights. 

On  the  other  hand,  tnere  was  no  special  attempt  made  to  preserve 
a quiet  laboratory  environment  during  testing.  Test  operators 
worked  in  the  same  room,  at  three  locations  separated  by  only  6 or  7 
meters,  and  could  hear  and  observe  each  other  at  work  if  they  wished 
to  do  so.  Other  users  of  the  laboratory  were  sometimes  present. 
Visitors  occasionally  wandered  about,  chatting  in  the  background. 
Phones  rang,  and  other  machine  noises,  particularly  the  high  speed 
printer,  were  obtrusive. 

There  was  no  time  pressure  on  the  test  operators  to  complete  the 
unloading  of  a truck,  or  to  keep  up  with  other  members  of  a work 
crew.  Each  man  worked  individually,  at  his  own  pace.  The  test 
operators  were  obviously  motivated  to  work  well,  however,  and  all 
were  diligent  in  their  task  performance.  Although  they  were 
encouraged  to  take  a break  after  each  test  session,  these  self- 
regulated  rest  periods  seldom  extended  for  more  than  5 or  10 
minutes. 

The  steady  rate  of  work  resulted  primarily  from  the 
conscientious  attitude  of  the  test  operators.  Other  contributing 
factors  were  implicit  competition  among  three  operators  working 
simultaneously,  and  a desire  to  finish  early.  Each  afternoon  when 
all  members  of  a test  group  had  completed  their  scheduled  test 
sessions,  they  were  given  an  opportunity  to  explore  other  data 
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processing  capabilities  available  in  the  DHAC,  including 
demonstrations  of  electronic  display  stations,  voice  input 
equipment,  bar-code  wand  reader,  and  optical  character  reader. 

Despite  such  interesting  "extracurricular"  activities,  and  the 
general  amenities  of  a laboratory  setting,  the  test  environment  was 
in  one  significant  way  more  demanding  than  the  actual  data  entry  job 
wculd  be.  At  the  truck  dock,  data  entry  would  be  performed  as  a 
sporadic  task,  with  occasional  intervals  of  waiting  for  the  next 
piece  of  cargo  to  be  unloaded  and  brought  into  view.  In  the  te3t 
environment,  the  label  for  the  next  piece  of  simulated  cargo  was 
always  waiting  in  the  stack  of  photographs,  so  that  the  data  entry 
task  required  continuous  concentration  of  attention.  There  is 
little  doubt  that  the  operators  were  tired  at  the  end  of  each  day's 
test  sessions. 

Observers 


Each  MAC  operator  worked  at  all  times  with  a MITRE  observer,  a 
different  person  for  each  data  entry  mode.  The  role  of  these 
observers  was  somewhat  broader  than  the  name  suggests.  The 
observers  were  responsible  for  training  test  operators  a3  well  as 
recording  performance,  and  also  served  as  MITRE  hosts,  answering 
questions,  providing  escort  to  the  cafeteria,  etc.  They  acted  more 
as  friendly  assistants  than  as  taskmasters. 

During  the  first  test  session  of  each  day,  the  observers 
instructed  the  operators  in  use  of  the  data  entry  modes,  offering 
general  advice  and  step  by  step  guidance  as  required.  Each  data 
entry  mode  had  its  own  special  features  which  influenced  how  the 
task  was  performed.  Operators  learned  the  necessary  procedures 
while  entering  data  for  the  first  few  pieces  in  the  test  load,  and 
thereafter  worked  on  their  own.  Throughout  the  first  test  session 
with  each  new  operator  the  observers  would  volunteer  comments  if 
they  saw  some  mistake  being  made,  and  were  prepared  to  answer 
questions  at  any  time.  During  subsequent  test  sessions  the 
observers  generally  did  not  comment  on  operator  performance,  except 
in  answer  to  specific  questions. 

The  observers  each  kept  a log  during  every  test  session,  noting 
any  apparent  errors  made  by  the  operators  and  any  points  of  seeming 
confusion  or  difficulty  with  data  entry  procedures.  Observers  were 
responsible  for  recording  the  beginning  and  end  times  for  each  test 
session.  For  the  two  modes  of  off-line  data  entry,  the  observers 
were  also  expected  to  record  the  time  required  for  various  different 
types  of  data  entry  sequences,  using  a stopwatch.  For  the  on-line 
terminal,  the  times  of  all  transactions  were  recorded  automatically 
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by  the  computer.  No  attempt  was  made  to  conceal  these  data 
recording  activities  from  the  test  operators.  The  operators  were 
'•nrcuraged  to  work  ouicklv  and  accurately,  and  they  understood  that 
tneir  performance  was  being  measured  in  both  respects. 

Performance  Measurement 

Measures  of  data  entry  speed  included  the  overall  time  required 
to  complete  each  session,  i.e.,  for  a 4tt-piece  test  load,  and  the 
time  consumed  by  different  types  of  data  entry  sequences.  Session 
time  was  recorded  by  the  observer  in  minutes.  Sequence  time3  were 
recorded  in  seconds.  Ihe  observers  used  a specially  formatted 
record  sneot  to  note  the  type  of  each  data  entry  sequence  as  well  as 
the  time  required.  Sequence  types  included  entry  ot  shipment  and 
piece  data  for  a "new"  TCN,  review  of  advance  shipment  data  and 
entry  of  piece  data  for  an  "old"  TCN,  and  entry  of  piece  data  only 
for  a TCN  whose  shipment  data  had  already  been  entered  or  reviewed. 

Within  a da^a  entry  sequence,  the  ooserver3  of  the  two  off-line 
modes  were  sometimes  able  to  record  the  time  required  for  individual 
transactions,  e.g.  , the  time  consumed  in  scanning  an  index  list,  in 
finding  tae  particular  page  in  a pack  of  checksheets,  in  reviewing 
shipment  data,  in  keying  a TCN,  etc.  For  the  on-line  terminal,  the 
controlling  computer  kept  a time  record  for  every  operator  input. 
This  record  was  stored  on  disk  for  later  analysis,  and  also  printed 
out  at  the  end  of  each  test  session.  It  was  thought  that  such 
transaction  analysis  could  prove  useful  in  the  future  when  test 
results  might  have  tc  be  interpolated  to  predict  performance  if  the 
data  items  to  be  entered  in  the  actual  job  setting  were  some  subset 
of  tnose  used  in  this  initial  test  program. 

A record  was  preserved  of  all  data  entries,  of  course,  and  for 
the  on-line  terminal  and  the  digital  recorder  a count  was 
subsequently  made  of  the  frequency  of  all  keystrokes,  for  data  entry 
itself  and  for  the  various  function  keys  used  to  control  data  entry. 
It  was  expected  that  such  records  would  help  the  interpretation  of 
any  measured  differences  in  data  entry  time  in  the  use  of  those  two 
devices.  The  data  records  were  preserved  in  digital  form  on  disk 
and  also  were  printed  out  for  visual  inspection. 

Accuracy  of  data  entry  for  the  on-line  terminal  was  determined 
by  programming  the  computer  to  compare  records  of  data  actually 
entered  with  a true  record  already  stored  for  each  test  load.  As  a 
result  of  this  comparison,  it  was  possible  to  note  those 
discrepancies  which  indicated  either  errors  in  entering  new  data  or 
failure  to  detect  errors  in  reviewing  advance  data. 
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For  the  digital  recorder,  the  measurement  of  accuracy  was  not 
quite  so  simple,  since  it  was  possible  for  the  test  operators  to 
make  errors  of  format  as  well  as  errors  in  data  content.  The 
operator  of  the  digital  recorder  had  to  push  special  function  keys 
to  indicate  that  the  next  symbol  group  he  entered  would  be  a TCN,  or 
a POD,  or  a consignee  code,  or  whatever.  If  he  made  an  error  using 
these  function  keys,  it  made  the  resulting  digital  record  difficult 
to  interpret.  When  the  computer  assembled  a data  file  from  the 
digital  record,  a correct  data  entry  might  be  associated  with  a 
wrong  item,  even  with  a wrong  shipment,  because  of  such  format 
errors. 

To  try  to  cope  with  this  problem,  the  digital  records  generated 
by  the  test  operators  were  submitted  to  computer  analysis  to  create 
an  initial  data  file  for  each  session,  which  wa3  then  printed  out 
for  inspection.  A number  of  common  format  errors  were  r oted  which 
had  resulted  in  garbled  data.  The  analysis  program  was  revised 
cope  with  some  of  the  most  damaging  format  errors,  and  then  used 
again  to  create  a new  data  file.  Those  refined  data  files,  still 
containing  some  unavailable  residual  losses  due  to  format  errors, 
were  then  compared  with  the  complete  true  records  for  each  test  load 
to  determine  data  entry  errors,  just  as  described  above  for  the  on- 
line terminal. 

For  the  checksheet  mode  of  data  entry,  it  was  not  possible  to 
develop  a sensible  measure  of  accuracy  in  the  context  of  this 
initial  test  program,  since  the  data  items  written  on  a checksheet 
are  not  yet  in  digital  form.  Use  of  a checksheet  implies  a 
subsequent  process  of  key  entry  which  would  itself  be  subject  to 
error.  Although  the  test  operators  were  cautioned  to  write  legibly, 
casual  inspection  of  the  resulting  checksheets  raises  some  question 
as  to  whether  their  data  could  in  fact  be  transcribed  accurately. 

It  is  planned  to  test  that  question  by  having  MITRE  operators 
enter  data  wonting  alternatively  from  the  scrawled  checksheets 
generated  in  initial  testing,  and  from  the  shipment  labels 
themselves.  The  results  of  that  follow-on  study  will  not  be 
available  until  later  this  year.  Any  data  errors  detected  in  that 
follow-on  study  will  be  examined  to  determine  whether  they  should  be 
attributed  to  faulty  recording  on  the  checksheet  or  faulty 
transcription,  although  as  a practical  matter  it  is  the  combined 
error  rate  which  would  be  important. 

Although  the  commission  of  data  recording  errors  using  the 
checksheets  cannot  be  assessed  here,  it  was  possible  to  mjasure  the 
operators'  succe33  in  detecting  errors  in  advance  shipment  data. 

All  checksheets  generated  during  this  initial  test  program  were 
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scanned  to  determine  how  many  of  the  known  errors  in  advance  data 
had  been  noticed  and  corrected  by  the  test  operators,  and  those 
results  are  documented  in  this  report. 


Operator  Evaluation 


In  addition  to  measuring  data  entry  performance,  it  was  also 
considered  important  that  the  test  operators  themselves  be  given  an 
opportunity  to  evaluate  the  data  entry  modes.  At  the  conclusion  of 
each  day's  test  sessions,  all  operators  were  asked  to  fill  out  a 
questionnaire  designed  to  record  their  opinions  of  the  particular 
data  entry  modes  they  had  used  that  day.  At  the  completion  of  all 
test  sessions,  on  the  third  day  of  testing,  the  operators  were  asked 
to  fill  out  a final  questionnaire  in  which  they  made  a comparative 
evaluation  of  the  three  data  entry  modes  they  had  used.  Copies  of 
tnese  two  questionnaires,  for  mode  evaluation  and  mode  comparison, 
are  included  in  Appendix  F to  this  report. 


SECTION  III 


HESULTS  - SPEED  OF  DATA  ENTRY 


If  a data  entry  task  at  the  truck  dock  is  superimposed  on  the 
primary  job  of  unloading  cargo,  it  is  important  to  ensure  that  data 
handling  does  not  delay  and  interfere  with  cargo  handling.  Thus  for 
any  data  input  mode  being  considered,  an  important  measure  is  sp***»d 
of  data  entry  at  the  truck  dock.  Measures  of  data  entry  speed 
obtained  in  this  initial  test  program  are  discussed  in  this  section 
of  the  report. 

Speed  of  data  entry  is  discussed  here  first  in  terms  of  the 
overall  time  per  session,  i.e.,  the  time  required  to  complete  a 48- 
piece  test  load.  Then  data  entry  is  analyzed  in  terms  of  the  time 
required  to  complete  characteristic  sequences  of  inputs  for  pieces 
of  cargo  with  and  without  advance  shipment  information.  Finally, 
this  section  provides  an  analysis  of  the  time  required  for  specific 
transactions . in  which  speed  of  data  entry  is  examined  for  the 
individual  items  on  a shipping  label. 

Because  the  time  required  is  strongly  dependent  on  the  amount  of 
data  to  be  entered,  this  discussion  of  data  entry  speed  i3  also 
accompanied  by  an  analysis  of  transaction  keying  for  the  two  input 
modes  Involving  use  of  keyboard  devices  - the  on-line  terminal  and 
the  digital  recorder. 


SESSION  TIME 

MITRE  observers  recorded  time  at  the  beginning  and  end  of  each 
of  the  72  te3t  sessions,  to  the  nearest  minute.  Those  session  times 
are  listed  in  Table  3-1.  Presented  in  this  form,  it  is  difficult  to 
discern  tne  relative  influence  of  the  several  experimental  variables 
because  of  their  overlapping  effects.  Aggregate  statistics  derived 
from  this  table  will  be  presented  in  the  paragraphs  that  follow,  in 
discussing  each  of  the  relevant  factors  involved. 

Statistical  significance  of  differences  in  data  entry  speed 
attributable  to  different  test  conditions  was  examined  through 
analysis  of  variance  of  session  times,  using  standard  computer 
software  developed  for  that  purposeP).  The  analysis  of 
effects  of  data  entry  mode,  shipraent-to-piece  ratio,  and  level  of 
advance  shipment  information,  is  summarized  in  Table  3-2.  A similar 
analysis,  in  wnich  session  times  were  re-ordered  to  examine  learning 
effects  by  day  and  by  session,  is  summarized  in  Table  3-3* 
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Table  3-1 


session  Time  for  48-Piece  Test  Loads 
ander  All  Teat  Conditions 


On-Line  T 

erminal 

6i&i  t_al. 

Recorder 

Checksheet 

Teat 

Load 

Day- 

Day  - 

Day- 

iterator  Factor  3 

Session 

lirng 

Session 

lime 

Session 

lime 

A 

HL 

2-3 

5d  mil 

n.  3-1 

76  min. 

1-1 

73  min 

Hh 

-1 

66 

-2 

46 

-2 

45 

LL 

-2 

42 

-4 

34 

-4 

37 

LH 

.4 

31 

-3 

2d 

-3 

32 

fc 

HL 

3-1 

53 

1-3 

49 

2-2 

40 

HH 

-3 

36 

-4 

3d 

-4 

32 

LL 

-4 

32 

-2 

34 

-3 

20 

LH 

-2 

2b 

-1 

69 

-1 

21 

C 

HL 

1-4 

60 

2-2 

60 

3-4 

42 

HH 

-2 

60 

-1 

62 

-3 

39 

LL 

-1 

74 

-3 

42 

-1 

35 

LH 

-3 

33 

-4 

37 

-2 

26 

u 

HL 

3-4 

60 

2-3 

44 

1-2 

38 

HH 

-1 

41 

-4 

36 

-4 

28 

LL 

-2 

32 

-1 

48 

-3 

23 

LH 

-3 

27 

-2 

29 

-1 

23 

E 

HL 

1-2 

62 

3-« 

48 

2-1 

45 

HH 

-4 

46 

-3 

45 

-3 

33 

LL 

-3 

43 

-2 

37 

-2 

27 

LH 

-1 

61 

-1 

47 

-4 

19 

F 

HL 

2-3 

68 

1-1 

96 

3-3 

46 

HH 

51 

-2 

56 

-1 

36 

LL 

-1 

64 

-4 

45 

-4 

31 

LH 

-4 

36 

-3 

42 

-2 

29 

Note : 

For  load  factors 

, HL  Indicates  a 

high  S/P 

ratio  ( .75)  and  a 

low 

level  of  advance 

shipment 

information  (.26);  LH  indicates  a 

low  S/P 

ratio  ( ,2b>  and  a high  level  of  advance  information  (.75);  etc. 
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Table  3-2 


Analysis  of  Variance  of  Session  Time  as  Influenced  by  Data  Entry 
Mode,  Shipment-to-Piece  Ratio,  and  Level  of  Advance  Shipment  Data 


Source  of  Variance 

d.f. 

Mean  Souare 

F 

Operators  (0) 

5 

463-5 

Conditions 

11 

Mode  (M) 

2 

1,404.5 

14.8* 

S/P  Ratio  (R) 

1 

3,556.1 

37.6* 

Advance  Data  (A) 

1 

1,404.5 

14. 8« 

M x R 

2 

2.93 

.03 

M x A 

2 

30.5 

.32 

R x A 

1 

162.0 

1.71 

M x R X A 

2 

159.5 

1.68 

Operators  x Conditions 

55 

94.7 

0 x M 

10 

111.1 

0 x R 

5 

143.2 

0 x A 

5 

83*3 

0 x M x R 

10 

85.7 

0 x M x A 

10 

77.6 

0 x R x A 

5 

129.7 

Residual 

10 

68.1 

•pC.001 
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Analysis  of  Variance  of  Session  Time  as  Influenced  by 
Day  of  Testing,  and  Session  within  Day 


Source  of  Variance 

d.f. 

Mean  Sauare 

F 

Operators  (0) 

5 

*63-5 

- 

Conditions 

11 

Day  < D) 

2 

162.9 

2.99 

Session  (S) 

3 

961.7 

6.22** 

D x S 

6 

202.0 

1.31 

Operators  x Conditions 

55 

151.6 

0 x D 

10 

299.5 

0 x S 

15 

125.7 

Residual 

30 

120.7 

Statements  of  statistical  significance  in  the  following  paragraphs 
are  based  on  these  two  analyses. 

Effect  of  Da ta  Entry  Mode 

The  use  of  different  modes  of  data  entry  resulted  in  fferences 
in  tne  length  of  test  sessions,  i.e.,  the  time  required  to  complete 
data  entry  for  a 48  piece  test  load.  Average  session  times  for  the 
three  modes  tested  are  listed  in  Table  3-4.  The  checksheet  was  the 
fastest,  witn  tne  otner  two  modes  about  the  same  in  overall  speed  of 
data  entry.  The  analysis  of  session  times  summarized  in  Table  3-2 
indicates  no  sigiifioant  interaction  effect  between  data  entry  mode 
and  any  other  test  variable. 


Table  3-4 

Average  Session  Time  for  Three  Data  Entry  Modes 


Data  Entry  Mode 
On-Line  Terminal 
Digital  Recorder 
Checks.-.oet 


Mean  Session  Time 
47.5  min. 

47.4 

34.2 


Ir.  actual  operations,  of  course,  nandwritten  data  on  the 
checksheet3  would  have  to  be  transcribed  in  some  way  to  digital 
form,  perhaps  by  keypunching,  thus  adding  to  the  total  time  required 
for  data  entry.  What  is  important  is  that  all  three  modes  seem  fast 
enough  for  use  at  the  truck  docks,  indicating  that  data  entry  could 
keep  pace  with  truck  unloading.  That  conclusion  will  be  discussed 
furtner  at  the  end  of  this  section,  after  all  results  on  speed  of 
data  entry  have  been  reported. 

Effect  of  Load  Factors 

Session  time  was  also  influenced  by  load  factors,  as  one  would 
expect  based  on  common  sense.  For  loads  with  a high  level  of 
advance  shipment  data,  fewer  new  data  items  have  to  be  entered,  and 
so  data  entry  is  faster.  For  loads  with  many  multi-piece  shipments 
(low  shipment-to-piece  ratio),  fewer  data  items  have  to  be  entered 
because  general  shipment  data  can  be  omitted  for  all  but  one  piece 
in  each  shipment;  and  again  data  entry  is  faster.  These  effects  are 
illustrated  in  Table  3-5,  which  presents  session  times  for  the 
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different  combined  load  factors  - h’gh  and  low  S/P  ratio,  and  high 
and  low  levels  of  advance  shipment  information  - averaged  over  all 
three  aata  entry  modes.  Tne  analysis  of  session  times  summarized  in 
TaDle  $-2  confirms  significant  effects  attributable  to  both  of  these 
load  factors,  but  indicates  no  significant  interaction  effect 
between  tnem. 


Table  3-5 

Average  Session  Time  for  Different  Test  Loads 
Load  Factors 

S/P  Hatio  Advance  Data 

riign  l .75)  Low  (.25) 

High  High  (.75) 

Low  (.25)  Low. 

low  High 


Mean  Session  Tine 

55.9  min. 

44.1 

38.9 

33. 1 


Learning  Effects 

Differences  in  average  session  time  from  day  to  day,  summarized 
in  Table  3-b,  were  not  great.  Although  there  seems  to  have  been  a 
tendency  for  the  test  operators  to  work  somewhat  more  quickly  from 
one  day  to  the  next,  the  analysis  of  session  times  presented  in 
Table  3-3  indicates  that  this  apparent  difference  does  not  quite 
achieve  statistical  significance  (p>.05).  


Table  3-6 

Average  Session  Time  from  One  Test  Day  to  Another 


Day  Mean  Session  Time 

1 47.8  min. 

2 42.0 

3 39.2 
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The  kinds  of  learning  which  might  result  in  day  to  day 
performance  changes  are  speculative.  The  operators  might  gradually 
become  accustomed  to  shipping  label  formats  and  the  general 
requirements  of  the  data  entry  task;  but  shipping  labels  were 
already  familiar  to  them.  The  operators  might  gradually  become  used 
to  the  general  test  environment  and  learn  to  concentrate  their 
attention  in  meeting  task  demands;  but  the  test  environment  although 
novel  was  not  particularly  stressful. 

More  notable  differences  in  time  were  observed  from  session  to 
session  witnin  each  day  of  testing,  as  summarized  in  Table  3-7.  It 
can  be  seen  tnat  the  first  test  session  of  the  day  went  fairly 
slowly  in  comparison  with  succeeding  sessions,  as  test  operators 
learned  the  procedures  for  the  particular  data  entry  mode  they  were 
using.  It  seems  evident  that  most  of  this  mode-specific  learning 
was  accomplished  within  the  first  tent  session,  since  little  further 
improvement  in  speed  can  be  observed  for  data  entry  in  subsequent 
sessions . 


Table  3-7 

Average  Time  for  Repeated  Sessions 

Using  the  Sane  Data  Entry  Mode 

Session  Mean  Session  Time 

1 53.8  min. 

2 41.1 


3 3n  ? 

U 37.8 


A special  analysis  of  session  time  was  conducted  to  determine 
whether  there  was  any  interaction  between  session  and  data  entry 
mode.  No  significant  interaction  effect  was  found.  One  might 
suppose  that  the  two  modes  involving  keyboard  data  entry  could  show 
greater  learning  effects  from  session  to  session  than  the  checksheet 
mode  which  only  involves  use  oT  paper  and  pencil.  The  results  do  in 
fact  match  that  expectation,  as  can  be  seen  in  Figure  9 where 
average  data  entry  time  is  plotted  against  session  separately  for 
the  three  data  entry  modes.  The  decrease  in  time  from  the  first  to 
second  session  does  seem  more  marked  for  the  on-line  terminal  and 
digital  recorder  than  for  the  checksheet  mode.  But  because  the 
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Figure  9.  Data  Entry  Time  as  a Function  of  Test  Session  for  Three 
Input  Modes 
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checksheet  is  generally  faster,  the  proportional  reduction  in  data 
entry  time  is  similar  for  all  three  modes,  thus  accounting  for  the 
failure  to  confirm  an  interaction  effect  in  variance  analysis.  Ey®n 
the  simple  checksheet  mode  provided  an  opportunity  for  learning,  if 
only  to  learn  better  how  to  shuffle  paper. 

Individual  Differences 

As  in  any  test  involving  human  performance,  there  were 
individual  differences  among  the  test  operators.  Average  session 
times  for  these  six  individuals  are  listed  in  Table  3-8.  It  can  be 
seen  that  the  fastest  operator  took  about  30  percent  less  time  than 
the  slowest.  (The  fastest  operator  is  not  necessarily  the  best  man 
for  the  job,  however,  unless  he  is  accurate  as  well.)  Since  the 
test  design  involved  repeated  leasurements  from  each  operator,  there 
can  be  no  valid  test  of  the  statistical  significance  of  individual 
differences  in  performance,  but  these  appear  to  be  of  about  the  same 
magnitude  as  differences  in  data  entry  speed  attributable  to  the 
controlled  teDt  variables. 

Table  3-8 

Individual  Differences  in  Speed  among  Six  Test  Operators 


Test  Operator 
A 
B 
C 
D 
E 
F 


Mean  Session  Time 
47.1  min. 

36.6 

47.5 

34.9 

41.9 
49-9 


Keystrokes 

To  some  extent  it  is  possible  to  account  for  differences  in  data 
entry  3peed  in  terms  of  different  demands  of  the  data  entry  task 
under  the  various  test  conditions.  Task  demands  could  be  estimated 
by  counting  data  entries,  or  perhaps  by  counting  keystrokes.  For 
the  two  modes  of  data  entry  which  produced  digital  records,  it  was 


possible  to  program  the  DHAC  computer  to  scan  those  records  for  each 
test  session  and  count  automatically  the  number  of  keystrokes  of 
various  kinds  wnich  were  used  to  accomplish  the  data  entry  task. 

Thus  there  is  available  a direct  and  convenient  statistic  reflecting 
the  amount  of  data  entered  for  test  loads  of  different  kinds. 

The  average  number  of  keystrokes  per  session  for  test  loads 
combining  different  load  factors  is  shown  in  Table  3—9 * listed 
separately  for  the  on-line  terminal  and  the  digital  recorder  to 
permit  comparison  of  those  two  data  entry  modes.  It  can  be  3een  in 
this  table  that  those  test  loads  which  resulted  in  shorter  session 
times  did  in  fact  require  substantially  less  data  input. 


Table  3-9 

Average  Keystrokes  per  Session  for  Different  Test  Load3  Using 
On-line  Terminal  and  Digital  Recorder 


Load 

Factors 

Data 

Entry  Mode 

S/P  Ratio 

Advance  Data 

On-Line  Terminal 

Digital  Recorder 

High  (.75) 

Low  (.25) 

1,939 

2,445 

High 

High  (.75) 

1,325 

1,663 

Low  (.25) 

Low 

1,297 

1,777 

Low 

High 

1,086 

1,567 

A comparison  of  performance  measures  for  different  test  loads  in 
Tables  3-5  and  3-9  certainly  indicates  a rough  correspondence 
between  time  required  and  data  entered.  But  this  correspondence 
does  not  exi3t  between  the  two  data  entry  modes,  which  were 
equivalent  in  time  required  (Table  3-4)  but  involved  quite  different 
levels  of  keying  activity.  In  effect,  users  of  the  digital  recorder 
were  required  to  make  more  keystrokes  to  accomplish  the  job,  but 
made  them  more  quickly.  Users  of  the  on-line  termimal  were 
presumably  constrained  to  a more  deliberate  pace  in  the  item-by-item 
sequence  of  interaction  with  the  computer.  The  net  result  for  these 
two  modes  of  data  entry  proved  to  be  equivalent  session  times 
achieved  in  different  ways. 

An  overall  count  of  keystrokes,  of  course,  is  not  an  exact 
measure  of  data  entered.  Some  keystrokes  are  needed  for  signalling 
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associated  with  the  control  of  entry  format,  and  for  error 
correction,  as  well  as  for  data  entry.  A more  detailed  analysis  of 
keying  actl* ity  for  the  on-line  terminal  is  summarized  in  Table  3-10 
and  for  the  digital  recorder  in  Table  3-11*  These  tables  show  the 
average  number  of  keystrokes  used  for  data  entry,  format  control  and 
error  correction,  for  different  test  loads. 

Comparing  keystroke  counts  for  the  two  entry  modes,  it  may  be 
seen  that  the  digital  recorder  required  on  the  average  about  400 
more  strokes  for  data  entry  than  the  on-line  terminal.  This 
difference  can  be  accounted  for  entirely  in  terms  of  the  different 
data  entry  procedures  used  in  this  initial  te3t  program.  The  user 
of  the  on-line  terminal  generally  did  not  have  to  enter  data  for  POE 
and  TAG,  instead  simply  confirming  computer-displayed  values  for 
those  two  data  items,  whereas  the  user  of  the  digital  recorder  had 
to  enter  all  items  completely. 

Comparing  the  two  data  entry  modes  in  terms  of  keystrokes 
required  for  format  control,  it  can  be  seen  that  the  overall 
proportion  was  fairly  similar  for  the  on-line  terminal  and  digital 
recorder,  38  and  32  percent  respectively.  This  "overhead"  for 
format  control  could  be  reduced  for  the  digital  recorder  by  redesign 
of  its  keyboard  to  permit  individual  item  entry  rather  than  line 
entry  of  groups  of  items,  as  recommended  in  Appendix  C,  with  savings 
of  pernaps  300  keystrokes  per  test  load. 

To  assist  in  comparing  the  two  data  entry  modes  in  terms  of 
keystrokes  required  for  error  correction,  a further  analysis  of 
error  correction  keying  is  presented  in  Table  3-12.  Here  it  can  be 
seen  that  BACKSPACE  was  by  far  the  most  common  procedure  employed 
for  error  correction  in  both  modes.  Considering  the  tremendous 
number  of  total  keystrokes  recorded  in  this  initial  test  program, 
the  relatively  slight  proportion  of  keying  required  for  error 
correction  is  evidence  of  conscientious  effort  by  the  test  operators 
to  maintain  accuracy  of  input  in  both  modes  of  data  entry. 


INPUT  SEQUENCE  TIME 

Tne  overall  time  per  session  gives  only  a rough  picture  of  the 
time  required  for  data  entry.  A more  detailed  view  would  take  into 
account  the  various  different  kinds  of  data  entry  sequences  which 
were  Included  in  this  initial  test  program.  Three  sequences  can  be 
usefully  distinguished: 

1.  The  first  piece  of  cargo  from  a "new"  shipment,  whose  TCN 
cannot  be  found  on  the  index  list,  for  which  the  test  operator  must 
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enter  the  full  TCN,  and  all  general  shipment  data  as  well  as  piece- 
specific  data. 

2.  The  first  piece  of  cargo  from  a shipment  with  advance  data, 
whose  TCN  is  on  the  index  list,  for  which  the  operator  can  enter  the 
short  index  designator,  and  review  general  shipment  data,  usually 
witnout  having  to  enter  any  changes,  before  entering  the  necessary 
piece-specific  data. 

3.  Any  extra  piece  of  cargo  in  a shipment  whose  general  data 
nave  already  been  entered  and/or  reviewed,  for  which  the  operator 
need  enter  only  the  index  designator  and  the  piece-specific  data. 

On  logical  grounds  one  would  expect  that  the  time  required  for 
data  entry  would  decrease  from  one  kind  of  sequence  to  the  next,  and 
this  indeed  proves  to  be  the  case.  The  average  times  required  for 
these  three  different  data  entry  sequences  are  listed  in  Table  3-13. 
for  the  three  modes  of  data  entry  used  in  this  initial  test  program. 
Table  3-13  also  includes  average  time  required  for  two  portions  of 
each  sequence,  the  first  part  involving  shipment  identification  plus 
entry  and/or  review  of  general  shipment  data,  and  the  second  part 
involving  simply  the  entry  of  piece-specific  data  which  is  common  to 
all  three  sequences.  Table  3-13  also  provides  for  each  sequence  and 
data  entry  mode  a parenthetic  indication  of  the  fastest  time 
recorded  in  this  initial  test  program. 

Several  additional  comments  will  aid  interpretation  of  these 
tabulated  performance  measures.  It  should  be  noted  that  this  table 
does  not  necessarily  permit  fair  comparisons  among  data  entry  modes. 
As  an  example,  from  these  tabulated  figures  one  might  suppose  that 
the  on-line  terminal  was  3omewhat  slower  to  use  than  the  digital 
recorder,  but  the  comparison  of  overall  session  times  has  shown 
those  two  devices  to  be  equivalent  in  speed.  The  seemingly 
anomalous  results  in  Table  3-13  can  be  accounted  for  by  several 
specific  differences  in  the  test  conditions.  For  the  on-line 
terminal  time  recording  was  automatic,  with  the  time  interval  for 
one  sequence  beginning  the  instant  the  previous  sequence  had  been 
completed.  Time  recording  for  the  digital  recorder  was  manual,  with 
the  observer  starting  his  stopwatch  only  when  the  operator  began 
work  on  a new  sequence,  which  could  account  for  1-2  seconds  of  the 
difference  in  measured  sequence  time  between  these  two  modes. 
Moreover,  the  observer  of  the  digital  recorder  occasionally  failed 
to  complete  his  time  recording,  when  he  was  distracted  by  offering  - 
advice  to  the  operator  or  answering  questions,  so  that  time  records 
were  sometimes  lost  for  that  subset  of  "trouble  trials"  which  may 
have  required  somewhat  longer  times  to  complete.  By  contrast, 
computer  recording  of  times  at  the  on-line  terminal  was  not 
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influenced  by  whether  its  operator  was  having  difficulties;  the  only 
times  eliminated  in  subsequent  analysis  of  the  computer  record  were 
for  those  few  instances  in  which  the  operator  elected  to  RESTART  a 
data  entry  sequence. 

On  the  other  hand,  Table  3-13  does  permit  a fair  comparison 
within  each  data  entry  mode  of  the  times  required  for  different 
entry  sequences.  As  an  example,  for  all  three  modes  it  can  be  seen 
tnat  the  sequence  involving  entry  of  complete  shipment  data  takes 
more  than  twice  as  long  as  the  sequence  in  which  advance  shipment 
data  can  be  reviewed  instead.  This  finding  suggests  that  if  all 
shipment  data  are  needed  for  processing  at  an  air  cargo  terminal, 
there  are  practical  benefits  in  permitting  review  and  confirmation 
of  advance  data  when  available,  rather  than  requiring  an  operator  to 
enter  all  data  directly  from  shipping  labels. 

It  can  be  seen  that  for  a given  data  entry  mode  the  entry  of 
pie?e  -specific  data  takes  approximately  the  same  time  in  all  three 
sequences  listed  in  Table  3-13*  An  exception  might  be  noted  for  the 
sequence  involving  review  of  advance  data.  For  a single-piece 
shipment  the  advance  data  would  include  a WT  and  CUBE  correct  for 
that  piece,  so  that  those  two  data  items  (considered  here  piece- 
specific  for  multi-piece  shipments)  could  be  simply  confirmed  during 
the  data  review  rather  than  entered.  As  a consequence,  the  average 
entry  time  for  piece-specific  data  was  measured  to  be  somewhat 
faster  for  sequences  involving  data  review. 

Average  values,  of  course  lo  not  tell  everything  about 
performance.  As  one  might  suppose,  the  time  required  to  complete  a 
data  entry  sequence  was  somewhat  variable  as  the  operators 
accomplished  their  job  more  quickly  for  some  pieces  of  cargo  than 
for  others.  This  kind  of  performance  variability  is  probably 
influenced  by  a number  of  factors,  including  such  things  as 
differences  in  label  legibility  as  well  as  fluctuating  attention  on 
the  part  of  the  test  operators.  For  the  on-line  terminal,  frequency 
distributions  of  sequence  times  were  developed  on  the  basis  of 
maenir.?  analysis  of  computer  records.  Probably  the  most  useful  way 
of  examining  such  distributions  is  in  terms  of  the  cumulative 
frequency  with  which  data  entry  is  accomplished  within  an  increasing 
time  limit.  Such  cumulative  distribution  curves  are  illustrated  in 
Figure  10  for  tne  three  kinds  of  data  entry  sequences  under 
discussion. 

The  curves  in  Figure  10  all  indicate  skewed  distributions  of 
sequence  antry  time,  lacking  that  symmetric  ogival  form  which 
characterizes  a balanced  "normal"  distribution.  In  effect,  in  such 
a task  it  is  not  possible  to  balance  the  occasional  instances  of 
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very  long  entry  times  with  an  equal  number  of  unusually  short  times. 
Given  such  skewed  distributions,  the  mean  value  is  not  necessarily  a 
characteristic  value.  For  all  three  data  entry  sequences  considered 
here,  more  tnan  60  percent  of  entries  were  accomplished  in  less  than 
the  average  times  reported  in  Table  3-13-  That  is  to  say,  the 
median  entry  times  were  lower  than  the  means  in  all  three  cases. 

With  distribution  curves  such  as  those  shown  in  Figure  10,  it  is 
possible  to  predict  expected  performance  within  any  selected  time 
limit.  Consider  the  rightmost  curve,  for  sequences  involving 
complete  entry  of  all  data  for  a new  shipment.  If  one  wished  to 
know  what  proportion  of  entries  were  accomplished  in  two  minutes  or 
less,  to  pick  an  arbitrary  example,  this  distribution  curve  permits 
an  estimate  of  75  percent.  The  curves  permit  estimation  in  reverse 
with  equal  ease.  As  an  example,  for  entry  only  of  piece-specific 
data  for  extra  pieces  in  a shipment,  what  is  the  time  limit  within 
which  90  percent  of  those  entries  can  be  accomplished?  The  leftmost 
curve  in  Figure  10  permits  an  estimate  of  45  seconds. 

Once  ar.  actual  data  entry  task  has  been  defined  to  support  air 
cargo  terminal  operations,  perhaps  a task  involving  entry  of  3ome 
critical  subset  of  shipping  label  data  items,  then  similar 
distribution  curves  could  be  developed  on  the  basis  of  field  testing 
to  predict  operator  performance  on  the  job.  Such  testing  should  be 
conducted  for  the  on-line  terminal  and  for  any  other  modes  of  data 
entry  considered  a3  feasible  alternatives  for  use  on  the  truck  dock. 


TKANSACTION  TIMING 

Data  entry  sequences  considered  at  a finer  level  of  detail  are 
comprised  of  the  discrete  transactions  by  which  individual  data 
items  are  entered  or  reviewed  For  the  two  off-line  modes  of  data 
entry  it  was  difficult  ti>  record  the  timing  of  separate 
transactions,  although  observer  notes  occasionally  included  timing 
lor  different  portions  of  a data  entry  sequence.  For  the  on-line 
terminal,  however,  the  controlling  computer  was  programmed  to  record 
the  time  of  every  operator  input,  and  so  a subsequent  analysis  at 

the  transaction  level  could  be  undertaken. 

\ 

There  are  several  ways  of  looking  at  the  results  of  transaction 
analysis.  Table  3-11*  lists  the  number  of  keystrokes  and  the  time 
required  for  different  transactions  of  data  entry  or  review, 
averaged  over  all  load  conditions  experienced  by  users  of  the  on- 
line terminal.  The  keystroke  count  includes  keying  needed  for 
sequence  control  as  well  as  for  data  entry,  so  that  keying  the  six- 
digit TAG,  for  example,  required  7.0  strokes  on  the  average  (i.e., 
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six  digits  plus  ENTEH).  These  counts  also  include  any  .eying  needed 
for  error  correction,  including  BACKSPACE,  so  that  some  of  the 
averages  are  slightly  higher  than  the  minimum  requirement:  e.g.,  2.1 
rather  than  2.0  for  PRIORITY,  4.1  rather  than  4.0  for  POD.  For  the 
"optional"  items  of  RDD,  PROJECT,  FROM  and  CONSIGNEE,  the  average 
keystroke  counts  included  occasional  entry  of  blank  items  and  so 
appear  lower  than  the  figures  logically  expected.  For  TCN/1NDEX  the 
average  of  6.1  keystrokes  includes  frequent  transactions  of  entering 
a 2-3  stroke  INDEX  as  well  as  entries  of  TCN  which  would  require  15 
strokes  (or  more  if  errors  were  made  and  corrected). 


The  transaction  timing  figures  in  Table  3-11*  are  generally 
related  to  the  nuraoer  of  keystrokes  involved,  as  one  would  expect. 

It  is  clear  that  data  review,  with  just  one  confirming  keystroke  per 
item,  is  faster  than  data  entry,  as  discovered  earlier  in  the 
analysis  of  sequence  timing.  It  should  be  noted,  however,  that 
tnese  tabulated  transaction  times  actually  represent  the  intervals 
between  one  ENTEH  input  and  the  next.  Consequently,  the  average 
time  figures  include  any  time  required  for  thinking,  checking, 
reading  displayed  prompts,  etc.,  as  well  as  keying  time  itself.  The 
mechanics  of  button  pushing  probably  require  no  more  than  one  second 
per  stroke.  None  of  tne  transaction  times  in  Table  3-11*  approach 
that  keying  speed  except  for  the  entry  of  WT  and  CUBE  which  are  loth 
snort  numeric  items. 

A different  way  to  look  at  transaction  records  is  shown  in  Table 
3-15.  Here,  an  analysis  of  data  entry  and  data  review  transactions 
is  presented  separately  for  test  conditions  involving  different  load 
factors.  (Tne  "Other"  category  in  this  table  represents  the  time 
spent  at  that  choice  point  in  the  control  sequence  offering  optional 
review  of  previously  entered  shipment  data.)  Instead  of  showing 
average  keying  and  time  required  per  item,  this  table  lists  the 
proportion  of  total  keying  or  total  time  required  to  enter  or  review 
each  data  item.  These  numbers  were  calculated  directly  from  the 
test  records  for  each  load  type,  and  were  not  derived  from  the 
overall  averages  in  Table  3-14. 

When  different  data  transactions  are  considered  in  terms  of 
their  proportional  demands,  certain  aspects  of  the  data  entry  task 
can  be  discerned  more  clearly.  One  can  see,  for  example,  how  the 
burden  of  tne  Job  shifts  from  data  review  to  data  entry  for  loads 
with  a lower  level  of  advance  shipment  information,  and  that  this 
snift  is  somewhat  different  for  loads  with  different  shipment-to- 
pieoe  ratios. 
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Table  3-15 


Proportion  of  Keystrokes  and  Time  Required  for  Different  Data  Transactions 
Using  tne  On-Line  Terminal  for  Different  Test  Loads 


Mean  Number  of 
Transactions 

Percent 

Keying 

of 

Total 

Percent 

Time 

of 

Total 

Load  Factors: 

HL 

HH 

LL 

LH 

HL 

HH 

LL 

LH 

HL 

HH 

LL 

LH 

Data  Entry 

TCN/INDEX 

51 

48 

4Q 

48 

26 

21 

19 

15 

32 

34 

30 

27 

HDD 

33 

25 

13 

7 

4 

3 

2 

1 

9 

7 

8 

4 

PROJECT 

35 

25 

15 

8 

4 

3 

2 

1 

3 

2 

2 

1 

PRIORITY 

28 

10 

10 

4 

3 

2 

2 

1 

2 

1 

1 

0 

FROM  (Consignor) 

31 

1b 

12 

6 

9 

5 

5 

2 

8 

4 

5 

2 

TO  (POE) 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

POD 

23 

10 

9 

4 

6 

3 

3 

2 

5 

3 

3 

2 

CONSIGNEE 

30 

17 

10 

6 

7 

4 

4 

2 

7 

4 

3 

2 

PIECE 

16 

Id 

46 

46 

2 

3 

7 

9 

2 

2 

7 

8 

TOTAL 

28 

9 

9 

3 

3 

1 

1 

1 

2 

1 

1 

1 

WT 

42 

26 

47 

48 

7 

7 

13 

16 

4 

3 

6 

8 

CUBE 

41 

25 

47 

48 

5 

5 

9 

1 1 

3 

2 

4 

6 

TAG 

2 

2 

3 

2 

1 

1 

1 

1 

0 

1 

1 

1 

DISP 

49 

49_ 

12 

48 

13 

19 

20 

24_ 

10 

12 

12 

17 

Totals: 

416 

280 

319 

278 

90 

*77 

88 

86 

87 

76 

83 

79 

Data  Review 
HDD 

5 

14 

4 

- 

0 

1 

0 

1 

1 

2 

1 

1 

PROJECT 

2 

13 

2 

5 

0 

1 

0 

1 

0 

1 

0 

1 

PRIORITY 

10 

28 

8 

10 

1 

2 

1 

1 

0 

2 

0 

1 

FROM  (Consignor) 

8 

22 

6 

8 

0 

2 

0 

1 

1 

2 

1 

1 

TO  (POE) 

37 

37 

17 

14 

2 

3 

1 

1 

2 

2 

1 

1 

POD 

10 

28 

7 

11 

0 

2 

1 

1 

0 

1 

0 

1 

CONSIGNEE 

7 

21 

7 

8 

0 

2 

1 

1 

1 

2 

1 

1 

PIECE 

30 

31 

3 

2 

2 

2 

0 

0 

2 

2 

1 

0 

TOTAL 

10 

28 

8 

10 

0 

2 

1 

1 

0 

2 

1 

1 

WT 

7 

23 

1 

1 

0 

2 

0 

0 

0 

1 

0 

0 

CUBE 

7 

23 

1 

1 

0 

2 

0 

0 

0 

1 

0 

0 

TAG 

47 

48 

48 

47 

3 

4 

4 

4 

4 

5 

5 

7 

Totals : 

180 

316 

112 

124 

8 

25 

9 

12 

1 1 

23 

; 1 

15 

Other 

12 

12 

_36 

_36 

1 

1 

3 

3 

1 

2 

5 

6 

Overall  Totals: 

608 

608 

467 

438 

Note:  For  load  factors,  HL  indicates  a high  S/P  ratio  (.75)  and  a 

low  level  of  advance  shipment  information  (.25);  LH  indicates  a 
low  S/P  ratio  (.25)  and  a high  level  of  advance  information 
(.75);  etc. 
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At  this  detailed  level  of  transaction  analysis  it  i3  possible  to 
use  the  numbers  in  Table  3-15  to  predict  the  effects  on  performance 
time  of  reconfiguring  the  data  entry  task  in  different  ways.  As  an 
example,  suppose  that  in  the  actual  data  entry  job  at  the  truck  dock 
it  was  nc.,  necessary  to  enter  or  review  the  "optional"  data  items  of 
HDD,  PROJECT,  FROM  and  CONSIGNEE.  Suppose  further  that  the  TO  (POE) 
item  could  be  omitted  on  the  assumption  that  it  would  be  the  same 
for  all  export  cargo  at  a given  air  terminal.  Suppose  finally  that 
the  identifying  TAG  on  each  piece  of  cargo  did  not  have  to  be  keyed 
but  instead  could  be  "wanded"  for  automatic  input  from  a machine- 
readable  label.  From  the  numbers  in  Table  3-15  it  can  be  estimated 
that  the  net  effect  of  all  of  these  changes  would  be  to  reduce 
keying  by  15  to  28  percent,  depending  upon  load  factors,  and  the 
time  required  for  the  job  by  21  to  3*1  percent. 

Examining  the  numbers  in  Table  3-15  it  is  evident  that  the 
single  most  time  consuming  transaction  in  this  job  is  entry  of  the 
TCN  or  INDEX,  which  by  itself  required  27  to  3**  percent  of  the  total 
time,  depending  on  load  factors.  If  the  technique  of  indexing  had 
not  been  used  to  shortcut  shipment  identification,  so  that  a full 
TCN  had  to  be  entered  for  every  piece  of  cargo,  then  this  one  type 
of  transaction  would  account  for  an  even  greater  proportion  of  the 
total  data  entry  time.  Tnis  observation  implies  that  the  single 
greatest  contribution  to  increased  efficiency  of  data  entry  on  this 
job  would  be  to  make  the  TCN  itself  machine-readable,  so  that  the 
operator  does  not  have  to  key  it,  but  instead  can  input  this  long 
symbol  string  by  the  easier  method  of  "wanding"  it.  Perhaps  the  TCN 
could  be  expanded  to  include  a piece  indicator,  and  this  combined 
identification  used  instead  of  the  extra  TAG  postulated  for  each 
piece  of  cargo  simulated  in  this  initial  test  program.  If  so,  the 
hardest  data  entry  transaction  measured  in  this  initial  test  program 
might  become  the  easiest  at  the  truck  dock. 


DATA  AVAILABILITY 

If  truck  unloading  could  proceed  without  pause,  with  no  rest 
breaks  fcr  the  work  crew  and  assuming  continuous  availability  of 
forklift  equipment  to  move  heavy  cargo,  the  time  required  might 
average  about  one  minute  per  piece.  In  actual  operations  at  the 
truck  docks  this  estimated  rate  is  seldom  achieved,  since  a number 
of  practical  problems  can  introduce  periodic  delays  in  the  unloading 
process.  Accepting  this  postulated  one-minute-per-piece  average  as 
a desirable  goal  nonetheless,  the  results  of  this  initial  test 
program  look  encouraging.  In  terms  of  session  time,  it  was  found 
that  all  three  data  entry  modes  permitted  data  input  at  an  overall 
average  rate  of  less  than  one  minute  per  piece  (Table  3-4). 
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It  is  true  that  several  aspects  of  test  performance  tend  to 
counter  this  encouraging  result.  For  the  most  difficult  test  loads 
(high  S/P  ratio,  low  advance  information)  data  entry  was  somewhat 
slower  than  one  minute  per  piece  on  average  (Table  3-5) • Also,  some 
individual  operators  worked  more  slowly  in  early  sessions  even  with 
relatively  easy  test  loads  (Table  3-1).  Furthermore,  the  timing  of 
data  entry  sequences  indicates  that  the  entry  of  complete  data  for  a 
new  shipment  always  took  longer  than  one  minute  for  the  on-line 
terminal  in  this  test  situation  (Figure  10)  and  so  conceivably  could 
impose  some  delay  on  unloading  at  the  truck  dock. 

In  tnis  initial  test  program,  speed  of  data  entry  was  measured 
for  a job  which  required  the  operators  to  enter  or  review  all  items 
from  a shipping  label  at  least  once  for  each  shipment.  In  actual 
data  entry  operations  at  the  truck  dock,  it  is  possible  that  the  job 
could  be  redefined  to  require  entry  only  of  some  smaller  subset  of 
critical  items,  as  suggested  above  in  the  analysis  of  transaction 
timing.  If  so,  actual  data  entry  performance  would  probably  be 
faster  than  that  measured  in  thi3  initial  testing,  and  one  data 
entry  operator  working  along  in  parallel  with  an  unloading  crew 
should  have  little  trouble  keeping  pace.  If  the  data  entry  task 
were  reduced  to  a minimal  set  of  data  items,  it  would  appear  that 
data  handling  need  not  slow  cargo  handling  whichever  mode  of  data 
entry  were  to  be  used. 

If  data  entry  modes  are  effectively  equivalent  in  providing 
adequate  speed  of  data  entry  at  the  truck  dock,  then  other  kinds  of 
measures  are  needed  to  determine  which  is  the  preferred  mode.  One 
such  measure,  of  course,  would  be  accuracy  of  data  entry,  as 
discussed  in  the  next  section  of  tnis  report.  Another  measure  which 
may  prove  important  in  overall  system  operations  is  the  availability 
of  entered  data  for  further  processing.  The  three  input  modes 
examined  in  this  initial  test  program  imply  considerable  differences 
in  data  availability. 

Data  inputs  made  using  an  on-line  terminal  are  immediately 
available  for  further  processing.  If  a piece  of  cargo  has  been  sent 
from  the  truck  dock  to  a particular  pallet  pit  or  storage  area,  that 
fact  would  be  known  immediately  in  the  general  data  processing 
system  used  to  support  air  cargo  terminal  operations. 

Data  inputs  entered  via  a digital  recorder  would  not  be 
available  for  further  processing  until  truck  unloading  is  completed. 
At  that  point,  the  digital  tape  record  could  be  read  into  the 
general  computer  system  in  a matter  of  only  a few  minutes.  In  the 
meantime,  however,  the  first  pieces  unloaded  from  a truck  may  wait 
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at  a pallet  pit  for  an  hour  or  more  before  their  data  records  are 
transmitted  to  the  computer. 

For  the  checksheet  mode  of  data  entry,  the  potential  deT  .■ 
between  data  recording  and  data  availability  is  longer  still  • ce 

a tru'0  has  been  unloaded,  its  ^ci-Ksheet  records  must  be 
trans  bed  into  digital  form  before  they  are  available  for  computer 
proce^  ig.  That  transcription  p.^cess  may  be  performed  by  skilled 
operators  working  under  relatively  good  conditions  in  a data 
handj.i..0  center,  but  even  assuming  continuous  availability  of 
personnel  (probably  unrealistic)  this  job  would  take  time,  perhaps 
another  30  to  60  minutes  per  load.  In  follow-on  testing  at  MITRE  it 
is  planned  to  measure  performance  of  operators  entering  checksheet 
data  into  a computer,  in  order  to  permit  better  estimation  of  just 
how  long  that  process  would  take. 

The  importance  of  delays  in  data  availability  can  be  evaluated 
only  in  the  context  of  a broader  analysis  of  data  handling  and  its 
effects  on  air  cargo  terminal  operations.  It  is  recommended  that 
such  a broader  analysis  of  system  performance  be  undertaken,  with 
emphasis  on  determining  what  are  acceptable  delays  in  data 
availability  following  data  entry  at  the  truck  dock.  By  comparing 
performance  requirements  established  in  that  analysis  with 
performance  capabilities  confirmed  by  testing,  it  should  be  possible 
to  decide  under  what  circumstances  the  digital  recorder  and 
checksheet  modes  can  be  used  as  effective  alternatives  to  on-line 
data  entry. 


SECTION  IV 


RESULTS  - ACCURACY  OF  DATA  ENTRY 


Fast  data  input  can  be  considered  useful  only  when  data  entries 
are  made  accurately.  If  data  entries  are  often  wrong,  or  perhaps 
accidentally  omitted,  then  the  data  entry  task  has  been  performed 
poorly.  Faulty  data  input  can  result  from  equipment  malfunctioning 
or  from  operator  error.  Whichever  the  cause,  some  modes  of  data 
entry  may  prove  more  accurate  than  others-. 

The  question  of  equipment  failure  can  be  dealt  with  briefly. 

The  on-line  terminal  worked  well  throughout  this  initial  test 
program.  No  flaws  in  terminal  or  computer  functioning  were 
detected.  For  the  checksheet  mode  of  data  entry,  there  was  no 
equipment  to  malfunction.  Sometimes  pencils  broke,  but  they  were 
replaced  as  needed. 

For  the  digital  recorder,  however,  some  difficulties  were 
encountered  in  hardware  functioning.  In  careful  scrutiny  of  the 
digital  records  produced  by  this  data  entry  mode,  a number  of  record 
gaps  were  detected,  which  were  caused  by  failures  in  "stepping"  the 
magnetic  tape  cassette.  Over  the  24  test  sessions,  a total  of  41 
tape  gaps  occurred,  varying  in  size  from  just  one  character  lost  to 
a sequence  of  several  hundred  missing  characters.  In  the  laboratory 
setting,  it  was  possible  to  retrieve  the  missing  data  by  a time- 
consuming  process  of  manual  re-keying  from  the  parallel  data  record 
produced  by  the  paper  strip  printer  on  the  digital  recorder.  In  any 
actual  data  entry  operation,  such  record  gaps  would  be  undetected 
until  coo  late  to  repair.  These  41  tape  gaps  would  have  resulted  in 
complete  data  loss  for  113  pieces,  partial  data  loss  for  13  more 
pieces,  and  wrong  data  recorded  for  10  pieces,  out  of  the  total  of 
1152  piece  records  entered. 

Such  data  loss  obviously  represents  an  unacceptable  hazard.  If 
digital  recorders  are  considered  for  operational  use,  some  means 
must  be  found  to  ensure  more  reliable  performance.  In  this  initial 
test  program,  most  of  the  tape  gaps  occurred  when  using  cassettes 
which  were  in  some  way  incompatible  with  the  digital  recorder.  When 
the  manufacturer's  own  cassettes  were  used,  such  gaps  occurred  only 
rarely.  It  should  be  noted  that  several  thousand  similar  . ecorders 
are  being  used  commercially,  apparently  without  serious  problems. 

The  question  of  operator  error  is  more  complicated,  and  will  be 
discussed  at  length  in  the  remainder  of  this  section.  The  topics  to 
be  considered  include:  the  frequency  and  nature  of  format  errors 
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made  by  the  test  operators;  errors  of  content  in  data  entry;  and 
operator  performance  in  the  detection  and  correction  of  errors, 
either  errors  made  during  data  entry  or  errors  in  the  pre-stored 
advance  shipment  data. 


FORMAT  ERRORS 

Suppose  that  an  operator  makes  a mistake  and  transposes  two 
characters  when  entering  a TON.  Such  an  error  in  content  would 
probably  not  be  detectable  in  a data  processing  system.  Tut  suppose 
that  the  operator  forgets  to  enter  any  TON  at  all,  or  en'.ers  a TCM 
with  too  few  or  perhaps  too  many  characters.  Such  errors  could  be 
detected  in  data  processing  because  they  violate  rules  which  define 
the  required  data  entry  format. 

For  the  on-line  terminal,  data  processing  follows  immediately  on 
data  entry,  so  that  format  errors  cannot  be  made.  Put  more 
accurately,  when  an  error  in  format  is.  made,  it  is  detected  by 
whatever  data  validation  procedures  have  been  included  in  the  on- 
line computer  software,  and  the  nature  of  the  error  is  signalled  to 
the  operator  so  that  he  can  correct  it  before  proceeding  to  the  next 
item  in  tne  data  entry  sequence.  A format  error  can  be  made,  but  it 
must  be  corrected  before  the  data  entry  sequence  can  be  continued. 

Format  errors  are  defined  in  terms  of  the  data  validation  checks 
which  are  applied.  For  the  on-line  terminal,  data  checks  and 
associated  error  messages  used  in  this  initial  test  program  can  be 
determined  from  review  of  the  control  sequence  outlined  in  Appendix 
B.  Other  modes  of  data  entry  could  involve  other  kinds  of  format 
errors,  as  discussed  below  for  the  digital  recorder. 

When  an  on-line  data  entry  sequence  is  fairly  straightforward, 
as  in  this  test  situation,  the  incidence  of  format  errors  will  tend 
to  be  low.  Errors  will  occasionally  be  made,  but  since  each  one  is 
brought  immediately  to  the  operator's  attention  he  will  learn  to 
avoid  tnem.  All  transaction  records  generated  while  testing  the  on- 
line terminal  were  analyzed  to  determine  the  frequency  of  format 
errors,  i.e.,  instances  in  which  operators  were  required  by  the 
computer  to  re-enter  a data  item.  Only  59  format  errors  were  found, 
as  summarized  in  Table  4— 1 . 

That  result  should  be  considered  in  terms  of  the  total  data 
entry  job  accomplished  in  this  initial  test  program.  Six  operators 
each  worked  in  four  sessions,  entering  data  for  48  pieces  per 
session,  with  3-14  data  items  per  piece  depending  upon  whether  data 
items  are  pre-stored  for  review  or  must  be  entered  from  the  label. 


Table  <4-1 


Data  Format  Errors  in  Use  of  the  On-Line  Terminal 


Type  of  Format  Error 


Data  Item 

Blank 

Too 

Short  Too  Long 

Other 

TCN/INDEX 

2 

19 

0 

1 (Note  2) 

Other  Items: 

RDD 

- (Note 

1)  0 

5 

- 

PROJECT 

• 

0 

0 

- 

PRIORITY 

0 

- 

- 

5 (Note  3) 

FROM  (Consignor) 

- 

7 

3 

- 

TO  (POE) 

0 

- 

- 

0 

POD 

4 

- 

- 

0 

CONSIGNEE 

- 

0 

1 

- 

PIECE 

2 

- 

- 

0 

TOTAL 

1 

- 

- 

0 

WT 

2 

- 

- 

0 

CUBE 

1 

- 

- 

0 

TAG 

1 

1 

0 

0 

DISP 

1 

- 

3 

*■ 



. 

— 

Totals: 

14 

27 

12 

6 

Total 

22 


5 

0 

5 

10 

0 

4 

1 

2 

1 

2 

1 

2 

4 

59 


Notes:  1.  indicates  that  no  data  check  was  made. 


2.  Attempted  entry  of  an  unassigned  INDEX. 

3.  All  five  instances  represent  attempts  to  enter  FROM 
data  when  asked  to  enter  PRIORITY. 
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This  represents  an  aggregate  total  of  approximately  8000  data  items 
entered.  Thus  tne  59  format  errors  observed  represent  only  a very 
small  proportion,  less  than  one  percent,  of  all  data  entries. 

There  are  not  enough  format  errors  listed  in  Table  4-1  to  reveal 
any  consistent  pattern  cf  mistakes,  except  for  the  apparent  tendency 
to  shorten  rather  than  lengthen  entry  of  the  TCN.  A 14-symbol  TCIJ 
constitutes  a data  string  which  exceeds  the  immediate  memory  span  of 
most  people.  Thus  entry  of  the  TCN  requires  glancing  back  and  forth 
from  label  to  keypad.  In  the  process,  some  symbols  occasionally  get 
lost. 

For  the  digital  recorder,  the  general  situation  is  auite 
different.  There  is  no  on-line  computer  to  detect  format  errors  of 
the  type  described  above.  Such  errors  might  be  detected  in 
subsequent  computer  processing  of  digital  records,  but  they  are 
irretrievable  in  the  sense  that  they  could  not  then  be  corrected. 

A more  significant  problem  in  use  of  the  digital  recorder  is 
that  there  is  no  on-line  control  of  the  data  entry  sequence.  An  on- 
line terminal  asks  Its  user  for  each  data  item,  and  thus  the 
controlling  computer  "knows"  what  item  is  being  entered.  Tne  user 
of  the  digital  recorder  must  remember  to  signal  which  data  item  he 
is  entering  next,  using  whatever  special  indicator  keys  are  provided 
for  tnat  purpose.  If  he  forgets  to  include  that  signal,  or  if  he 
makes  a mistake  and  records  the  wrong  signal,  a correct  data  entry 
may  be  lost  or  garbled  in  subsequent  data  processing  of  the  digital 
record. 

Such  format  errors  in  signalling  the  sequence  of  data  entry  may 
nave  disastrous  effects  on  the  accuracy  of  data  records.  Simple 
omission  of  a single  key  stroke  could  lose  the  complete  data  record 
for  a piece  of  cargo  and  produce  wrong  data  entries  for  the 
preceding  piece.  Care  must  be  taken  in  equipment  design,  in 
operator  training,  and  in  the  logic  used  for  analyzing  digital 
records,  to  reduce  the  likelihood  of  such  format  errors  and  minimize 
their  consequences  insofar  as  possible. 

Digital  records  generated  during  this  initial  test  program  were 
scanned  to  determine  the  nature  and  frequency  of  format  errors.  A 
total  of  183  errors  were  detected,  involving  improper  data  entry 
procedures  of  various  different  kinds.  Of  these  1 o 3 errors,  24  were 
minor  in  nature  and  would  have  no  consequences  in  subsequent  data 
processing.  The  remaining  159  format  errors,  using  a 
straightforward  method  for  computer  processing  of  the  digital 
records,  would  have  the  effect  of  losing  all  data  for  76  pieces, 
creating  wrong  data  records  for  85  pieces,  adding  false  data  records 
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for  3 imaginary  pieces,  losing  data  for  98  miscellaneous  items  and 
recording  wrong  data  for  94  other  items.  A more  detailed  discussion 
of  format  errors  using  the  digital  recorder  is  presented  in  Appendix 
E to  this  report. 

The  computer  software  used  to  analyze  digital  records  was 
revised  in  various  ways  to  try  to  make  it  "smart"  enough  to  correct 
some  of  tne  more  serious  format  errors.  To  consider  one  example, 
occasionally  a test  operator  using  the  digital  recorder  would  forget 
to  key  the  special  indicator  when  beginning  to  enter  the  TCN  for  a 
new  piece  of  cargo.  The  effect  of  that  simple  error  in  any 
straightforward  data  analysis  scheme  would  be  to  overlay  subsequent 
data  entries  as  "corrections"  to  data  items  just  entered  for  the 
immediately  preceding  piece,  thus  storing  a complete  set  of  wrong 
data  for  that  piece  and  losing  any  record  of  the  new  piece.  To  try 
to  remedy  this  situation,  the  data  analysis  software  was  revised  to 
include  a search  for  each  piece  to  determine  whether  the  entry  of 
disposition  code  (properly,  the  last  data  item)  was  directly 
followed  by  a long  string  of  symbols  (assumed  to  be  a new  TCN)  in 
which  case  a TCN  indicator  was  inserted  before  data  analysis 
continued.  Otner  kinds  of  format  errors  were  corrected  in  other 
ways.  A description  of  various  correction  routines  is  included  in 
Appendix  E. 

Using  this  improved  data  analysis  software,  a number  of  format 
errors  in  the  digital  records  still  remained  uncorrected,  and  indeed 
uncorrectable.  The  net  result  was  89  uncorrected  format  errors 
(including  6 introduced  by  the  "correction"  routines)  resulting  in 
complete  data  loss  for  14  pieces,  wrong  data  records  for  21  pieces, 

7 false  records,  and  lost  or  wrong  data  for  131  other  individual 
data  items.  The  effects  of  these  residual  format  errors  are  listed 
in  greater  detail  in  Table  4-2.  It  is  obvious  that  there  are  limits 
to  how  well  machine  analysis  can  counteract  operator  error. 

For  the  checksheet  mode  of  data  entry,  analysis  of  accuracy 
poses  a problem.  The  data  items  written  on  a checksheet  may  contain 
errors,  and  still  further  errors  may  be  introduced  in  tne  process  of 
transcribing  written  checksheet  data  into  digital  form.  Data  items 
may  have  been  recorded  correctly  but  illegibly,  as  suggested  by  the 
sample  cnecksheets  illustrated  in  Appendix  D.  For  the  checksheet 
mode,  any  sensible  analysis  of  accuracy  must  take  into  account  such 
potential  problems  of  data  transcription. 

There  is  little  point  in  looking  at  the  checksheets  and  trying 
to  guess  what  the  error  rate  of  data  transcription  would  be.  That 
must  be  measured  empirically.  A follow-on  study  is  now  being 
designed  to  measure  the  speed  and  accuracy  of  data  transcription 
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Table  4-2 


Consequences  of  Format  Errors  in  Use  of  the 
Digital  Recorder 


Number 


Number 


Data  Item 

Lost 

Wrong 

TCN/INDEX 

14 

piece 

records 

21 

piece 

Other  Items: 

RDD 

0 

items 

(Note  3) 

7 

items 

PROJECT 

4 

0 

PRIORITY 

3 

0 

FROM  (Consignor) 

14 

0 

TO  (POE) 

2 

7 

POD 

2 

11 

CONSIGNEE 

4 

8 

PIECE 

b 

1 

TOTAL 

0 

3 

WT 

5 

7 

CUBE 

9 

2 

TAG 

6 

14 

DISP 

12 

5 

Totals: 

66  other 

items 

65  other 

Notes:  1.  After  computer  correction  of  digital  records, 
89  residual  format  errors  remained,  some  of 
which  had  multiple  consequences. 


2.  9 TCN's  were  too  short  as  entered  and  3 too  long. 


3.  Lost  or  wrong  data  records  caused  by  faulty 
entry  of  TON  are  not  included  in  the  remainder 
of  this  tabulation,  which  represents  the 
effects  of  miscellaneous  other  format  errors. 
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from  the  checksheets  generated  in  this  initial  test  program,  and 
results  of  that  study  will  be  reported  when  available.  Meanwhile, 
in  this  analysis  of  initial  test  results  no  attempt  has  been  made  to 
estimate  the  accuracy  of  checksheet  records,  either  in  terms  of 
format  errors  or  errors  in  data  content. 


ERRORS  IN  DATA  CONTENT 

Some  errors  cannot  be  detected  by  discrepancies  in  data  format 
and  cannot  be  attributed  to  faulty  data  entry  procedures,  but  result 
from  other  simpler  kinds  of  mistakes.  An  operator  may  misread  a 
data  item  when  transcribing  from  a shipping  label,  or  he  may  see  the 
item  correctly  but  hit  a wrong  key  when  entering  the  data.  If  his 
rtake  does  not  invalidate  the  format  of  the  item  entered,  the 
■.•ror  cannot  be  detected  by  machine  analysis.  The  operator  may  not 
\o.ice  a data  item  on  the  shipping  label,  perhaps  a consignee  code, 

; - i so  neglect  to  enter  it.  If  that  item  is  not  required  by  format 
cu  iventions,  then  its  absence  will  not  be  noticed  in  any  machine 
analysis  of  tna  data  record.  Errors  of  these  kinds,  which  do  not 
\ ..''late  format  rules,  are  referred  to  here  simply  as  errors  in  data 
content. 

To  discover  such  content  errors  in  the  data  record  is 
potentially  a difficult  task.  In  this  initial  test  program,  the 
technique  used  to  identify  content  errors  involved  several  steps. 
First,  a true  record  of  all  shipping  label  data  was  stored  in  the 
DrIAC  computer.  When  testing  was  completed,  the  computer  was 
programmed  to  compare  each  digital  data  record  actually  input  by  an 
operator  with  the  true  record  for  that  test  load.  The  results  of 
that  comparison  were  then  printed  out  with  any  discrepancies  flagged 
to  call  attention  to  them.  These  printed  records  were  then  scanned 
to  determine  what  content  errors  of  different  kinds  had  occurred  and 
what  was  their  effect  on  the  accuracy  of  the  final  data  records 
produced  during  testing. 

For  the  on-line  terminal,  the  input  data  could  be  examined 
directly  in  this  fashion.  For  the  digital  recorder,  it  was 
necessary  first  to  process  the  data  record  through  several  screening 
routines  designed  to  compensate  for  certain  format  errors,  as 
described  above  and  in  Appendix  E,  before  the  final  records  could  be 
scanned  for  content  errors.  For  the  checksheet  mode,  the  data  had 
not  been  entered  in  digital  form,  and  so  no  such  machine  analysis  of 
content  errors  was  feasible.  As  argued  earlier,  any  sensible 
analysis  of  checksheet  errors  must  await  a follow-on  study  in  which 
checksheet  data  are  transcribed  to  digital  form,  and  resulting 
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errors  can  be  assessed  to  determine  which  should  be  attributed  to 
faulty  data  entry  and  which  to  mistakes  in  data  transcription. 

Insofar  as  errors  in  data  content  are  concerned,  the  present 
analysis  permits  comparisons  only  between  the  on-line  terminal  and 
the  digital  recorder.  A summary  of  content  errors  for  these  two 
modes  of  data  entry  is  presented  in  Table  4-3,  showing  the  number  of 
errors  made  for  different  items  on  the  shipping  label.  For  the  on- 
line terminal,  102  concent  errors  were  noted,  31*  involving  wrong 
entries  for  TCN  or  INDEX,  plus  56  other  items  entered  wrongly  and  12 
items  omitted  altogether.  For  the  digital-  recorder,  134  content 
errors  were  noted,  distributed  as  shown  in  the  table. 

Errors  of  omission,  resulting  in  missing  items,  must  presumably 
be  attributed  simply  to  inattentiveness  on  the  part  of  the  test 
operators,  momentary  lapses  in  their  generally  effective  level  of 
performance.  It  is  only  tne  optional  data  items  (PROJECT, 

CONSIGNEE,  etc.)  which  are  included  here.  Omission  of  a required 
data  item  was  classified  earlier  as  a format  error  rather  than  an 
error  in  data  content,  since  such  omissions  could  be  detected  in 
subsequent  processing  of  the  data  record. 

Errors  involving  wrong  data  content  may  result  from  more 
complicated  causes.  A categorization  of  such  wrong  entries  i3 
presented  in  Table  4-4.  The  most  common  category,  of  course,  is  the 
substitution  error,  in  which  one  symbol  is  wrongly  entered  in  place 
of  another.  Substitution  errors  accounted  for  more  than  80  percent 
of  wrong  data  content  in  both  the  on-line  terminal  and  digital 
recorder  entry  modes.  More  than  half  of  these  substitution  errors 
involved  confounding  of  0 (zero)  and  0 (the  letter),  presumably  a 
perceptual  problem.  Many  other  substitution  errors  involve  adjacent 
numbers  (1  for  2,  2 for  3,  7 for  8,  etc.)  or  adjacent  letters,  which 
may  reflect  keying  errors. 

Other  types  of  errors  producing  wrong  data  can  be  identified: 
transposition,  where  two  symbols  are  entered  in  reverse  order; 
omission,  where  a symbol  has  been  lost  in  data  entry;  additions, 
where  an  extra  symbol  has  been  entered;  displacement,  where  one  data 
field  is  accidentally  entered  for  another  (e.g.,  CUBE  instead  of 
DISP);  etc.  Such  mistakes  are  quite  infrequent  for  conscientious 
operators,  and  are  usually  accepted  as  representing  an  inescapable 
residual  level  of  human  error. 

The  confounding  of  0 and  0,  however,  is  a common  mistake  which 
is  induced  by  tne  circumstances  of  this  data  entry  task.  Most 
shipping  labels  are  typed  or  printed  in  such  a way  that  those  two 
symbols  look  identical  and  can  be  distinguished  only  on  the  basis  of 
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Table  4-3 


A Comparative  Count  of  Errors  in  Data  Content 


On-Line 

Terminal 

Digital 

Recorder 

Data  Item 

Missing 

Wrong 

Missing 

Wrong 

TCH/ INDEX 

- (Note 

1)  34  (Note  2) 

- 

28  (Note 

Other  Items: 

RDD 

0 

1 

0 

0 

PROJECT 

0 

9 

1 

14 

PRIORITY 

- 

0 

— 

1 

FROM  (Consignor) 

4 

4 

11 

16 

TO  (POE) 

- 

0 

1 

POD 

- 

1 

• 

0 

CONSIGNEE 

8 

11 

15 

16 

PIECE 

- 

2 (Note  3) 

- 

5 (Note 

TOTAL 

- 

0 

• 

0 

WT 

- 

2 

- 

5 

CUBE 

- 

1 

4 

TAG 

- 

2 

- 

4 

DISP 

- 

23 

- 

13 

Totals: 

IT 

90 

~TT 

107 

Notes:  1.  inchoates  items  whose  absence  would  be  termed 

a format  error  rather  than  an  error  in  data  content. 


2.  Resulting  in  2 piece  records  lost,  42  wrong. 

3.  Resulting  in  2 piece  records  lost,  2 wrong. 

4.  Resulting  in  4 piece  records  lost,  36  wrong. 

5.  Resulting  in  5 piece  records  lost. 
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Table  4-4 


Error  Categorization  for  Wrong  Data  Items 


On-Line  Terminal 

Digital 

Recorder 

TCN/INDEX 

Other  Items 

TCN/INDEX  Other  Items 

Substitution  errors 

31 

42 

25 

64 

0/0 

9 

8 

6 

11 

0/0 

13 

7 

13 

25 

Other 

9 

27 

6 

28 

Transposition  errors 

1 

0 

2 

3 

Omissions  (too  short) 

- 

5 

- 

3 

Additions  (too  long) 

- 

1 

- 

3 

All  other  types 

O 

C. 

8 

1 

6 

Totals: 

~ 

"56* 

28 

79 

\ 

\ 
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meager  contextual  clues.  It  is  somewhat  surprising  to  note  that 
none  of  the  test  operators  ever  asked  for  guidance  in  this  regard, 
perhaps  having  (unjustified)  confidence  that  they  could  guess  which 
symbol  was  intended  on  the  shipping  label.  But  the  consequences  of 
a wrong  guess  in  any  straightforward  data  processing  system  can  be 
severe  if  the  wrong  symbol  has  been  entered  in  a TCN,  as  for  example 
keying  DJ0B...  for  a shipment  whose  TCN  actually  begins  DJOB....  In 
this  initial  test  program,  such  confusions  of  0 and  0 accounted  for 
it  1 wrongly  input  TCN's,  which  resulted  in  wrong  data  records  for  54 
pieces  of  cargo. 

Fortunately,  there  are  several  potential  solutions  which  could 
entirely  eliminate  this  particular  type  of  error,  or  at  least  its 
consequences,  in  the  entry  of  TCN's.  One  possible  solution  would  be 
procedural,  to  require  that  all  shipping  labels  be  typed  or  printed 
in  such  a way  as  to  distinguish  between  0 and  0.  Because  of  human 
unreliability,  that  approach  would  probably  be  only  partially 
successful.  Another  procedural  solution  would  be  to  enforce  a ban 
on  the  use  of  the  letter  0 in  TCN  designators.  That  approach  looks 
feasible  but  might  be  a bother  to  implement.  Perhaps  the  best 
solution  is  one  which  could  be  accomplished  in  the  data  processing 
system  itself.  A screening  routine  could  be  devised  to  convert  all 
0's  into  0's  (or  vice  versa)  in  any  entered  TCN  before  comparing  it 
with  pre-stored  TCN  records  which  are  converted  in  the  same  way  for 
purposes  of  comparison.*  Thus  whichever  symbol  had  been  entered  by 
the  operator,  0 or  0,  a match  could  be  found.  Some  such  approach  as 
this  might  well  prove  useful  in  actual  MAC  data  handling  operations. 

Having  alluded  several  times  in  the  foregoing  discussion  to  the 
occurrence  of  operator  error,  it,  may  be  worthwhile  to  examine  tnat 
question  more  specifically.  Individual  differences  in  accuracy 
among  the  six  test  operators  car  be  discerned  in  the  summarized 
results  presented  in  Table  4-5.  In  this  table,  the  number  of 
content  errors  (of  all  kinds)  made  by  each  operator  is  shown  for 
each  of  the  two  data  entry  modes  for  which  results  have  been 
analyzed.  For  purposes  of  comparison,  a similar  count  of  format 
errors  made  by  users  of  the  digital  recorder  is  also  included.  (For 
the  on-line  terminal  no.  format  errors  were  possible  in  the  final 
data  records.)  Statistically  significant  differences  among 
individual  operators  are  confirmed  with  respect  to  error  frequency 
in  all  three  categories  of  error.  All  six  test  operators  made  some 
errors,  of  course,  but  two  of  these  operators  (designated  he.  e D and 
F)  seem  to  have  been  more  error-prone  than  the  others. 


* This  solution  was  suggested  at  MITRE  by  Warren  E.  Anderson. 
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Table  4-5 


Differences  in  Accuracy  among  Test  Operators 


On-Line  Terminal 

Digital 

Recorder 

Test 

Content 

Content 

Format 

ODerator 

Errors 

Errors 

Errors 

A 

9 

11 

5 

B 

16 

11 

9 

C 

11 

10 

15 

D 

26 

40 

19 

E 

16 

15 

5 

T+ 

c 

24 

47 

30 

Totals : 

102 

TFT 

83 

Individual 
Di f ferenees 

•x* 

13.6 

61.9 

33.9 

P 

<.05 

<.001 

<•001 
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It  is  interesting  that  the  four  "careful"  operators  made  no  more 
errors  of  data  content  when  using  the  digital  recorder  than  when 
working  with  the  on-line  terminal,  whereas  the  two  "careless" 
operators  made  relatively  more  content  errors  witn  the  digital 
recorder.  In  applying  human  engineering  principles  to  the  design 
of  equipment,  it  is  sometimes  argued  that  a device  which  is  well 
designed  can  be  used  better  by  more  people  and  so  tends  to  reduce 
variability  in  individual  performance.  If  that  is  a sound  argument, 
then  the  present  results  would  suggest  that  it  is  the  on-line 
terminal  which  was  the  easier  device  to  use  in  this  test  situation. 
The  test  operators  themselves  generally  concurred  in  this  judgment 
when  making  thier  own  explicit  evaluations  of  the  different  data 
entry  modes,  as  discussed  in  Section  V of  this  report. 

Although  a comparison  between  data  entry  modes  has  been 
maintained  throughout  the  preceding  discussion  of  input  accuracy,  it 
is  probabiy  useful  to  recapitulate  the  basic  results  here.  Table 
6 presents  an  overall  summary  of  the  frequency  and  consequences  of 
data  entry  errors,  including  errors  of  data  content  for  the  on-line 
terminal,  and  errors  of  both  content  and  format  for  the  digital 
recorder.  In  terms  of  data  content,  error  rate  wa3  quite  low  with 
consequences  about  the  same  for  both  devices.  The  primary 
difference  between  these  two  data  entry  modes  is  the  additional 
damage  to  data  records  caused  by  format  errors  in  using  the  digital 
recorder,  errors  of  the  kind  which  were  corrected  under  computer 
guidance  when  tne  same  operators  worked  with  the  on-line  terminal. 


ERROR  DETECTION  AND  CORRECTION 

Throughout  their  data  entry  task,  of  course,  the  test  operators 
were  aware  of  the  possibility  of  error  and  took  steps  to  correct 
those  errors  which  were  detected.  The  operators  were  in  fact 
successful  in  correcting  many  errors,  both  errors  which  they  made 
themselves  during  data  entry  and  errors  in  pre-stored  data  which 
they  detected  during  review  of  advance  shipment  information.  These 
two  categories  of  error  correction  are  discusser  in  the  following 
paragraphs. 


Data  Entry 


Error  correction  was  possible  in  all  three  data  entry  modes  used 
in  this  initial  test  program.  In  working  with  the  checksheets,  the 
operators  occasionally  erased  and  re-wrote  mistaken  data  entries. 

In  the  other  two  modes  of  data  entry,  the  operators  used  the 
BACKSPACE  capability  and  sometimes  other  procedures  for  correction 
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Table  4-6 


Consequences  of  Data  Entry  Errors  Using  On-Line  Terminal 
and  Digital  Recorder 


On-Line  Terminal 

Digital 

Recorder 

Content 

Content 

Format 

Errors 

Errors 

Errors 

Number  of  errors 

102 

134 

83 

Piece  records  lost 

4 

9 

14 

Piece  records  wrong 

44 

36 

21 

Other  data  items  missing 

12 

27 

66 

Other  data  items  wrong 

56 

79 

65 

84 


of  entry  errors,  as  discussed  earlier  in  the  analysis  of  transaction 
keying  presented  in  Section  III  of  this  report. 

In  terms  of  what  can  be  learned,  it  does  not  seem  worth  the 
effort  to  scan  the  off-line  records  produced  by  test  operators, 
looking  for  corrected  entry  errors,  i.e.,  erasures  on  the 
checksheets  or  special  symbols  of  various  kinds  in  the  printed 
outputs  of  the  digital  recorder.  Such  a scanning  process  would  be 
tedious  and  itself  highly  subject  to  error.  For  the  on-line 
terminal,  however,  it  proved  a relatively  easy  matter  to  program  the 
DHAC  computer  to  scan  its  records  of  the  interactive  data  entry 
process  and  print  out  a machine  analysis  of  which  data  items  had 
been  corrected  using  wnat  means.  The  results  of  that  analysis  are 
summarized  in  Table  4-7. 

It  should  be  emphasized  that  the  numbers  in  Table  4-7  count  only 
those  data  entry  errors  which  were  corrected  by  the  operators 
themselves,  and  do  not  include  format  errors  which  the  operators 
were  required  to  correct  by  prompting  from  the  on-line  computer. 
Altogether,  188  data  entry  errors  were  corrected  spontaneously  by 
the  operators.  Host  of  these  corrections  (88  percent)  were 
accomplished  using  the  BACKSPACE  capability. 

The  relative  frequency  of  error  corrections  for  the  various 
different  data  items  entered  at  the  on-line  terminal  illustrates  one 
pattern  of  interest,  namely,  that  more  corrections  were  required 
entering  alphabetic  or  mixed  alphanumeric  items  than  entering  simple 
numeric  items.  A fair  comparison  in  this  regard  might  be  that 
between  DISP  (54  errors  corrected)  and  WT  (21  corrections).  A 
similar  pattern  can  be  discerned  in  the  distribution  of  uncorrected 
errors  shown  previously  in  Table  4-3.  Presumably  the  greater 
difficulty  of  alphabetic  entries  is  related  to  the  double-keying 
technique  required  to  enter  letters  with  this  particular  on-line 
keypad.  By  comparison,  straight  numeric  entries  are  easier. 

The  overall  frequency  of  error  correction  using  the  on-line 
terminal  (188)  compares  favorably  with  the  number  of  wrong  entries 
which  went  uncorrected  (90),  indicating  conscientious  effort  by  the 
test  operators  to  maintain  accuracy  in  their  data  inputs.  It  is 
probable  that  the  same  effort  was  also  applied  by  the  operators  when 
working  with  the  other  two  data  entry  modes. 

Data  Review  I 

In  addition  to  entering  new  data,  the  test  operators  were  also 
instructed  to  review  all  items  of  pre-stored  shipment  data  and  to 
correct  tnose  items  which  were  discrepant  with  the  actual  shipping 
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Table  4-7 

Self  Detection  and  Correction  of  Data  Entry  Errors 
by  Operators  Using  the  On-Line  Terminal 


Number  of  Data  Errors  Corrected  Using 


Data  Item 

BACKSPACE 

CANCEL 

BACKUP 

TCN/ INDEX 

39 

5 

1 

j 

Other  Items: 

RDD 

1 

PROJECT 

4 

PRIORITY 

2 

FROM  (Consignor) 

9 

TO  (POE) 

S'f 

POD 

14 

3 

CONSIGNEE 

11 

1 

! '■ 

PIECE 

4 

TOTAL 

1 

WT 

21 

2 

CUBE 

5 

1 

TAG 

1 

DISP 

54 

9 

» 

Totals: 

166 

21 

— 

f 

label  information.  All  errors  in  the  pre-stored  data  were  either 
known  in  advance  or  determined  during  subsequent  analysis  of  data 
records  produced  during  testing.  Operator  inputs  were  compared 
against  the  list  of  known  errors  in  pre-stored  data  to  determine 
which  errors  had  been  detected  and  corrected.  This  comparison 
process  was  performed  with  computer  assistance  for  the  digital  data 
records. 

Data  review  and  correction  procedures,  of  course,  were 
available  to  the  test  operators  in  all  three  data  entry  modes.  For 
the  operators  using  checksheets,  their  error  correction  procedure 
was  to  cross  out  wrong  items  of  pre-stored  data  and  write  in  the 
correct  item  from  the  shipping  label,  and  also  to  write  in  any  items 
missing  from  the  pre-stored  data.  All  checksheet  records  were 
scanned  to  determine  how  many  of  the  known  wrong  and  missing  items 
of  pre-stored  data  were  in  fact  corrected  during  the  test  sessions. 
Thus  for  this  particular  aspect  of  performance,  an  analysis  of 
accuracy  is  available  for  the  checksheet  mode  of  data  entry  as  well 
as  for  the  on-line  terminal  and  digital  recorder. 

The  results  of  this  analysis  are  presented  in  Table  4-8.  It  can 
be  seen  that  detection  and  entry  of  missing  items  of  pre-stored  data 
was  quite  good  in  all  three  modes  of  data  entry,  averaging  93 
percent  overall.  Operator  performance  in  noticing  and  correcting 
wrong  items  of  pre-stored  data  was  quite  good  using  the  on-line 
terminal  (90  percent)  but  somewhat  poorer  in  the  digital  recorder 
and  checksheet  modes  (53  and  67  percent,  respectively).  Chi-square 
analysis  of  error  detection  frequency  for  wrong  items  of  pre-stored 
data  confirms  the  statistical  significance  of  this  observed 
difference  in  performance  among  the  three  input  modes  ("X2  = 14.9; 
pC.01) . 

. .... It  seems  apparent  that  it  is  easier  to  notice  a missing  item  of 

data  than  a wrong  item.  A missing  item  represents  a blank  in  the 
data  record  where  there  is  not  a blank  on  the  shipping  label.  A 
wrong  item,  on  the  other  hand,  may  differ  from  its  correct  (label) 
version  by  only  one  symbol,  and  so  would  be  harder  to  notice.  The 
effectiveness  of  the  on-line  terminal  in  correcting  wrong  pre-stored 
data  can  probably  be  attributed  to  its  particular  procedure  of 
displaying  items  one  by  one  for  review,  thus  focussing  the 
operator's  attention  at  least  momentarily  on  each  item.  In  the 
other  two  modes  of  data  entry,  the  operator  had  to  scan  the 
checksheet  printouts  to  review  pre-stored  data.  Under  those 
circumstances,  the  operators'  self-adopted  scanning  procedures  were 
not  so  effective  as  that  discipline  imposed  by  the  on-line  computer. 
The  observer  for  the  digital  recorder  mode  did,  in  fact,  note 
several  occasions  in  which  data  review  seemed  quite  cursory. 
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Tfble  4-8 


Detection  and  Correction  of  Errors  in  Review 


of 

Pre-Stored 

Data 

Pre-Stored  Data  Items 
Missing 

Pre-Stored  Data  Items 
Wrong 

Data  Entry  Mode 

Corrected 

Not 

Co  'rectert 

Corrected 

Not 

Corrected 

On-Line  Terminal 

22 

1 

38 

4 

Digital  Recorder 

17 

2 

16 

14 

Checksheet 

27 

2 

28 

14 

88 


sSBEaesrt'i” 


Perhaps  it  should  be  noted  here  that  this  superior  detection  of 
wrong  pre-stored  data  items  using  the  on-line  terminal  was  r.ot  a 
result  of  any  sort  of  data  validation  checks  applied  by  the 
computer.  All  pre-stored  data  items  were  assumed  to  have  passed 
data  screening  procedures  before  storage,  so  that  no  data  checks 
were  applied  during  the  review  process  unless  a revision  was  made. 

If  a wrong  item  was  confirmed  as  correct  by  tne  operator,  it  was 
retained  in  storage  without  any  further  checking  by  the  sequence 
control  software.  In  this  initial  test  program,  detection  of  wrong 
items  in  data  review  was  the  responsibility  of  tne  man,  not  the 
machine. 

Correction  of  errors  in  pre-stored  data  was  influenced  in  some 
measure  by  the  type  of  data  item  in  which  the  error  occurred. 

Results  to  support  that  conclusion  are  summarized  in  Table  4-9.  In 
this  tabulation  it  can  be  seen  that  failures  to  detect  errors 
occurred  relatively  more  frequently  for  what  have  been  termed 
"optional"  data  items  than  for  required  data  items.  In  this  initial 
test  program,  four  data  items  were  optional  in  the  sense  that 
sometimes  they  were  blank  and  no  data  entry  was  required:  RDD, 
PROJECT,  FROM  (consignor  code),  CONSIGNEE.  These  four  data  items 
account  f^r  most  instances  (75  percent)  of  failure  to  correct  wrong 
or  missing  pre-stored  data.  It  seems  likely  that  the  test  operators 
habitually  gave  somewhat  less  attention  to  data  items  which  did  not 
appear  consistently  in  the  data  entry  task  sequence. 

This  observation  suggests  that  to  help  ensure  reliable 
performance  j.n  real  data  entry  job,  it  would  be  wise  to  select 
just  a subset  of  shipping  data,  a minimal  number  of  items  wnose 
entry  would  be  required  for  every  piece  of  cargo.  This  suggestion 
for  redesign  of  tne  data  entry  job  applies  primarily  to  use  of  off- 
line modes  of  data  entry  such  as  the  digital  recorder  cr  the 
checksheet,  where  consistency  of  input  is  solely  a responsibility  of 
the  operator.  For  the  on-line  terminal,  where  the  computer- 
generated control  sequence  is  continuously  available  to  guide  its 
user,  the  combination  of  man  and  machine  can  maintain  performance 
accuracy  even  when  the  task  contains  sporadic  elements. 

Correction  of  errors  in  pre-3tored  data,  just  like  all  otner 
aspects  of  performance,  varied  somewhat  from  one  test  operator  to 
another.  Differences  among  operators  in  error  Jetection/correction 
are  shown  in  Table  4-10,  summed  over  all  three  data  entry  modes. 

For  wrong  items  of  pre-stored  data,  correction  rates  ranged  from  33 
to  92  percent,  for  the  "worst"  and  "best"  operators.  Individual 
differences  illustrated  in  Table  4-10  seem  to  correspond  roughly  to 
differences  in  error  commission  discussed  earlier  and  summarized  in 
Table  4-5,  indicating  that  each  operator  was  fairly  consistent  in 
his  ability  to  maintain  accuracy  in  these  different  aspects  of  job 
performance. 
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Table  4-9 


Error  Correction  for  Different  Categories  of  Pre-Stored  Data 


Pre-Stored  Data  Items  Pre-Stored  Data  Items 
Missing Wrong 


Data  Item 


Not  Not 

Corrected  Corrected  Corrected  Corrected 


RDD  16 
PROJECT  15 
PRIORITY  3 
FROM  (Consignor)  2 
TO  (POE) 

POD 

CONSIGNEE  10 
PIECE  5 
TOTAL  5 
WT  5 
CUBE  5 


2 6 0 

0 9 6 

0 7 2 

1 17  7 

3 0 

9 3 

2 14  10 

0 

0 - - 

0 9 3 

0 8 1 


To tals 


66 


5 


82 


32 


Table  4-10 


Differences  in  Error  Detection/Correction  among  Test  Operators 


Pre-Stored  Data  Items 
Missing 


Pre-Stored  Data  Items 
Wrong 


Test 

Operator 

A 

B 

C 

D 

E 

F 


Corrected 

16 

12 

14 

11 

7 

6 


Not 

Corrected 

2 

0 

1 

2 

0 

0 

5 


Corrected 

17 

22 

13 

13 

12 

5 

82 


Not 

Corrected 

4 

3 

7 

7 

1 

10 

32 


Totals:  66 
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SECTION  V 


RESULTS  - OPERATOR  EVALUATION 

At  the  end  of  each  day's  testing,  the  operators  were  asked  to 
record  their  evaluation  of  the  data  entry  mode  they  had  just  used, 
completing  a questionnnaire  designed  for  that  purpose.  At  the 
conclusion  of  all  three  days  of  testing,  the  operators  were  asked  to 
compare  the  three  data  entry  modes  directly,  completing  another 
questiom.aire.  The  results  of  these  operator  evaluations  are 
presented  in  this  section. 


MODE  EVALUATION 

The  exact  questionnaire  format  used  for  the  evaluation  of  data 
entry  modes  at  the  end  of  each  day  is  illustrated  in  Appendix  F to 
this  report.  The  questionnaire  sequence  will  be  followed  here  in 
reporting  operator  evaluation. 

General  Comment 

The  first  question  asked  the  operators  to  record  their  general 
reaction,  to  discover  which  aspects  of  their  experience  seemed  most 
notable  to  them: 

After  a day's  experience  using  this  particular  input  mode  to 
enter  shipping  data,  what  Is  your  general  reaction?  Which 
aspects  of  the  job  go  well,  and  which  poorly?  What  advantages 
does  this  data  input  mode  provide?  What  disadvantages? 

Operator  responses  are  presented  here  verbatim  for  the  three  data 
entry  modes.  - - 

On-Line  Terminal 

Operator  A.  "I  like  this  system  better.  It  is  simple  and  would  be 
a great  help.  It  helps  in  keeping  information  straight  and  it  is 
very  easy  to  correct  your  mistakes.  You  don't  have  any  equipment  to 
get  in  your  way  except  may  be  one  small  cord.  It  is  a fast  and  very 
efficient  system." 

Operator  B.  "This  is  the  best  out  of  the  three  systems  we  tested. 

It  is  an  excellent  system  for  checking  your  mistakes.  It  could  be 
used  also  when  building  up  pallets  of  cargo.  This  system  would 
eliminate  a lot  of  paperwork.  It  is  a comfortable  machine  to  work 
with  and  it  is  easy  to  learn  how  to  use  it." 
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Operator  C.  "I  think  this  little  machine  will  speed  the  processing 
up  and  make  the  cargo  move  faster  through  the  system.  I think  in  my 
opinion  that  there  wasn't  any  bad  aspects  towards  the  machine.  The 
good  aspects  that  I could  see  was  it  speeds  up  the  system  about 
twice  as  fast." 

Operator  D.  "The  on-line  terminal  device  was  a very  good  handling 
device  but  the  device  get's  right  warm  in  the  palm  of  your  hand, 
other  than  that  I like  it  the  best." 

Operator  E.  "My  general  reaction  is  that  the  On  Line  Terminal  is 
fantastic  makes  the  job  much  quicker.  It  would  cut  down  paper  work 
to  a dragstic  measure  it  would  allow  the  cargo  to  be  moved  faster  no 
need  to. wait  for  the  Mateo  men  for  wright  up.  In  ray  opinion  there 
are  no  disadvantages." 

Operator  F.  "One  disadvantage  is  that  it  gets  hot.  Correcting 
prestored  data  wasn't  too  good." 

Digital  Recorder 

Operator  A.  "This  device  did  not  impress  me  to  much.  I would  say 
it  is  the  most  complicated  of  the  three  we  tested.  One  of  the 
disadvantages  is  that  you  would  have  to  carry  the  whole  machine  with 
you  or  have  a desk  where  you  can  operate  it  from.  If  you  did  this 
it  would  mean  one  less  man  on  a crew  for  physical  work." 

Operator  B.  "This  mode  becomes  easier  as  you  do  it  more.  The  more 
advanced  documentation  you  have  the  easier  it  would  be.  It  is  much 
quicker  using  this  machine  than  having  to  go  running  through  a lot 
of  different  bills.  If  this  was  to  be  used  it  would  need  to  be  used 
in  all  areas  of  the  freight  system,  if  not  then  I can't  really  see 
what  it  would  eliminate  as  far  as  checking  a truck  off." 

Operator  C.  "I  think  this  mode  would  be  better  if  you  didn't  have 
to  carry  all  of  it  with  when  you  were  working.  I think  it  would  be 
good  because  you  could  always  go  back  to  the  tapes  if  you  ever  had 
any  trouble.  This  mode  provides  faster  movement  of  cargo." 

Operator  D.  "Your  hand  gets  tired  from  holding  it.  Other  than  that 

I liked  it." 

Operator  E.  "My  reaction  as  before  with  the  other  modes  is  the  same 

with  the  digital  recorder.  All  aspects  of  tne  job  go  well.  It 

allows  the  paper  work  two  be  written  up  much  faster  then  the  one 
presently  being  used." 


Operator  F.  "I  see  no  disadvantages  other  than  it  size  and  it 
advantages  is  speed." 

Checksheet 

Operator  A.  "It  is  an  easy  method  to  learn  and  U3e.  No  one  should 
have  any  trouble  with  this  method.  I really  couldn't  judge  on  the 
advantages  and  disadvantages  of  this  type  of  system.  Over  all  it  is 
a very  simple  system." 

Operator  B.  "As  in  all  modes  of  input,  the  more  advanced 
information  you  have  the  easier  it  is.  It  is  pretty  easy  to  check 
for  mistakes  this  way.  The  biggest  advantage  is  that  all  your  work 
would  be  combined  in  one  checksheet  rather  than  with  a lot  of 
different  bills.  It  doesn't  get  in  your  way.  A person  would  have 
no  worries  about  damaging  a piece  of  expensive  equipment.'' 

Operator  C.  "I  think  this  checksheet  is  good  but  I don't  tnlnk  it 
would  be  as  good  as  the  other  two  machines.  The  way  it  is  layed  out 
makes  is  go  well  and  fast.  The  advantages  is  that  is  makes  the 
systo.i  easyer  and  quicker." 

Operator  D.  "1  had  a little  trouble  finding  some  of  the  numbers  but 
I liked  the  checksheet  as  it  was.  The  trouble  I had  was  with  the  0 
the  line  through  the  zero  _s  what  gavs  me  the  trouble.  At  a quick 
glance  it  looked  like  an  8 not  a zero." 

Operator  E.  "Again  this  method  used  in  my  opinion  is  faster 
better  than  the  one  presently  being  used.  It  would  allow  all 
personnel  to  be  used  instead  of  Mateo  write  ups.  It  also  again  very 
simple  to  be  trained  on  10  minutes." 

■t  

Operator  F.  "To  much  paper  work,  other  than  that  it  works  well." 

What  conclusions  can  be  drawn  from  these  comments?  The  general 
response  seems  favorable  to  all  three  modes.  Several  drawbacks  are 
cited,  but  these  are  usually  characterized  as  minor  in  comparison 
with  the  recognized  advantages  of  each  data  entry  technique.  It  is 
evident  that  as  the  operators  used  different  modes  of  data  entry, 
from  day  to  day,  they  began  to  draw  comparisons  between  one  mode  and 
another.  In  that  process,  the  on-line  termin:l  seems  the  mode 
regarded  most  favorably. 

Overall  Rating 

The  next  question  asked  the  operators  to  *-ate  their  overall 
evaluation  of  the  data  entry  mode: 
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Check  one  of  the  boxes  on  the  right 
to  indicate  your  overall  evaluation 
of  this  input  mode  for  the  task  of 
entering  shipment  data. 


On-Line  Terminal  (OT) 

Digital  Recorder  (DR) 

Checksheec.  (CS) 

This  tabulation  shows  the  number  of  operators  who  chose  each 
rating.  The  on-lir.e  terminal  is  confirmed  as  the  most  popular  mode, 
with  a unanimous  ■’excellent"  rating.  If  a numeric  value  of  4 is 
assigned  for  each  "excel. ent"  rating,  3 for  "good",  2 for 

"adequate",  1 fo-  "hair",  and  0 for  "poor'’,  then  an  average  numeric 

rating  for  each  input  mode  can  be  derived:  4.0  for  the  on-lii.e 
terminal,  2.7  for  the  digital  recorder,  and  3.0  for  he  checksheet. 

"hese  derived  numeric  ratings  are  included  here  and  in  subsequent 

tabulations. 

Specific  Performance  Ratings 

The  next  question  asked  the  operators  to  rate  specific  aspects 
of  task  performance: 
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Detecting  errors  in  pre-stored  data OT 

4 

1 
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Correcting  errors  made  during  data  entry.. OT 
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The  on-lire  terminal  is  rated  comparatively  well  in  all  aspects 
except  detection  of  errors  in  advance  (pre-stored)  shipment  data. 

The  test  operators,  of  course,  did  not  have  available  to  them  the 
performance  results  described  earlier  in  this  report  which  confirmed 
that  the  on-line  terminal  was  clearly  the  best  mode  in  terms  of 
detecting  wrong  items  of  pre-stored  data. 

If  the  ratings  listed  above  for  specific  aspects  of  performance 
are  averaged,  the  resulting  aggregate  ratings  again  confirm  a 
preference  for  the  on-line  terminal  (3.8)  in  comparison  with  the 
digital  recorder  (2.9)  and  the  checksheet  (3.2),  These  derived 
aggregates  correspond  closely  to  the  overall  ratings  assigned  by  the 
test  operators  in  the  previous  question,  and  indicate  a reasonable 
consistency  in  operator  responses. 

Comments  on  Learnlng/Tralnlng 

Next,  the  operators  were  asked  for  comments  on  learning  to  use 
the  data  entry  mode  they  had  Just  experienced: 

What  are  your  general  impressions  of  your  experience  in  learning 
how  to  use  this  input  mode?  Which  aspects  were  difficult?  How 
could  your  training  have  tjeen  improved? 

On-Line  Terminal 

Operator  A.  "Over  all  it  is  the  best  system  I have  seen  yet.  There 
are  know  aspects  which  are  difficult." 
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Operator  h.  "It  isn’t  a difficult  mode  of  input  to  learn.  Anyone 
couii  learn  to  use  it  in  a day’s  experience." 

Operator  C.  "what  really  impressed  me  was  the  way  you  could  always 
go  back  to  something  and  refer  to  it  and  3ave  time  by  that." 

Operator  D.  "The  device  wasn't  to  hard  to  learn  how  to  use  and  1 
nad  a little  difficulty  hitting  the  correct  button.  I don't  think 
that  my  training  could  have  been  improved." 

Operator  L.  "There  were  no  difficult  aspects  of  learning  how  to 
operate.  It  Just  takes  about  ten  minutes  to  learn  and  get  use  to 
it.  The  training  period  was  conducted  in  an  excellent  fasion." 

Operator  F.  "it  wa3  easy" 

Digital  Recorder 

Operator  A.  "My  impression  is  that  it  would  be  unpracticle  in  my 
field  of  work.  Over  all  it  is  easy  to  operate  and  doesn't  take  long 
to  learn  how  to  operate  it." 

Operator  B.  "It  takes  time  to  get  used  to  it.  Nothing  was  very 
difficult.  It  is  easy  to  learn  hew  to  use  it.  The  best  way  to 
train  someone  on  this  machine  is  the  way  I learned.  You  can  learn 
more  quickly  by  doing  it  first  hand  and  learning  by  your  mistakes." 

Operator  C.  "I  thought  it  was  difficult  to  learn  the  key  board 
because  of  so  many  Keys.  I it  is  better  for  a guy  to  have  someone 
tell  him  what  the  key3  are  for  then  more  less  learn  on  his  own  just 
like  did  I think  that  i3  the  best  way  of  training  someone." 

Operator  D.  "I  thought  it  wa3  very  easy  at  the  end  of  it  but  I did 
have  some  trouble  getting  use  to  it." 

Operator  E.  "There  were  no  difficulty  in  learning  total  training 
time  was  fantastic.  So  was  the  training." 

Operator  F.  "it  was  easy  to  learn" 

Cnecksheet 

Operator  A.  "My  impression  is  that  this  is  a quick  and  easy  input 
mode." 

Operator  b.  "It  was  easy  for  me  because  I work  with  boxes  with 
these  labels  all  the  time.  I'm  used  to  looking  at  all  the  writing 
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and  I can  cheek  it  fairly  easy.  Someone  who  la  inexperienced  would 
have  a more  difficult  time  at  first.  As  in  anything,  the  more  you 
do  it  the  easier  it  gets." 

Operator  C.  "It  was  very  easy  to  learn  how  to  use.  I didn’t  find 
any  that  were  difficult.  I think  training  was  good." 

Operator  D.  "The  only  difficulty  1 had  was  look  at  a quick  glance 
the  numbers  were  a little  close." 

Operator  tr.  "It  was  very  simple  to  learn  how  to  use  this  specific 
input  mode.  The  training  again  was  conducted  superbly." 

Operator  F.  "it  was  easy  to  learn  and  use" 

The  generally  positive  comments  reported  here  probably  reflect 
several  factors  - the  high  morale  of  the  operator  groups,  their 
pride  in  accomplishment,  and  a desire  to  compliment  the  MITRE 
observers  who  had  provided  their  training  and  guidance. 

Rating  of  Learning  Ease 


admit  any  learning  difficulty,  even  when  some  specific  difficulties 
were  noted  in  their  previous  comments.  It  is  interesting  that  no 
poor  ratings  were  given  the  digital  recorder,  which  in  the  opinion 
of  MITRE  observers  was  in  Tact  a difficult  data  entry  mode  to  learn. 
Presumably  the  operators  themselves  remained  largely  unaware  of 
their  frequent  format  errors  In  using  that  device,  errors  which 
reflected  in  3oae  degree  a failure  to  learn  skilled  performance,  as 
well  as  simple  lapses  of  attention. 


99 


In  * ha*  4*  ? 

l he  r.oxt  qansMon  asked  the  operators  to  rate  their  Interest  In 

the  jo-  : 


I'or.si  2*»r:ng  your  experience  with  thi3 
i np--t  ^ rate  your  interest  in  the 
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The  nigh  interest  rating  for  the  on-line  terminal  probably 
reflects  a "pinball"  effect  - the  novelty  of  this  initial  exposure 
of  tne  operators  to  direct  interaction  with  a computer.  By 
contrast,  the  simple  paper  shuffling  of  the  check3heet  mode  was 
rated  the  least  Interesting;  and  the  digital  recorder,  an 
unrespor.si vt  device,  received  an  intermediate  interest  rating. 


Perform, a:ice  Rating 


Tne  next  que;  lien  asked  the  operators  to  rate  their  performance, 
as  a consistency  check  on  similar  ratings  made  ealier: 


Considering  3peed  and  accuracy  of  data 
processing,  and  any  other  factors  you 
think  important,  how  would  you  rate  your 
performance  using  this  input  mod*:? 


Again,  the  on-line  terminal  received  more  favorable  ratings.  It 
may  be  noted  that  the  performance  ratings  here  seem  somewhat  lower 
tnan  ratings  given  in  response  to  similar  questions  earlier  in  the 
questionnaire.  The  key  is  probably  the  qualifier  "your 
performance".  The  operators  may  have  been  influenced  by 
considerations  of  modesty  in  rating  their  own  performance  of  the 
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data  entry  task,  In  contrast  to  earlier  questions  which  were  phrased 
to  request  a more  detached  rating  of  mode  capability. 

Equipment  hating 

The  next  question  asked  the  opei ator3  to  evaluate  the  equipment 
used,  i.e.,  the  physical  implementation  of  each  mode: 

Hew  would  you  rate  the  physical  


equipment  used  in  this  input  mode? 
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Again  there  is  the  suggestion  of  a "pinball"  effect,  with  higher 
ratings  given  the  two  data  entry  modes  actually  involving  hardware 
devices.  That  is  to  say,  it  is  probable  that  the  operators  were 
influenced  by  the  novelty  of  equipment  used,  rather  than  the 
adequacy  of  mode  implementation  considered  in  relation  to  task 
requirements. 

Other  Comments 

A final  question  gave  the  operators  a chance  to  note  any  further 
observations  on  topics  not  already  covered  in  the  questionnaire: 

What  other  comments  can  you  make  concerning  the  equipment,  or 
any  other  aspect  of  the  test  situation? 

On-l.lne  Terminal 

Operator  A.  "I  think  it  is  a very  good  system." 

Operator  B.  "This  mode  seems  excellent  to  me.  I would  highly 
recommend  it's  use  in  any  freight  terminal." 

Operator  C.  "It  is  a very  modern  and  fast  way  to  get  the  job  done." 
Operator  D.  {None} 

Operator  E.  "The  heat  emitted  by  hand  commuter  can  be  cut  down 
some.  But  overall  I am  amased  with  this  equipment." 
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operator  F.  "Try  to  cool  it  down  it  gets  to  hot  in  your  hand." 
Digital  Hecorder 


Operator  A.  I None! 

Operator  B.  "It  wouldn't  take  any  great  length  of  time  in  teaching 
someone  to  use  this  machine.  It  would  be  easier  than  trying  to 
train  someone  all  the  different  type3  of  paperwork  there  is." 

Operator  C.  "I  think  in  my  own  opinion  that  this  should  be  looked 
into  farther  in  maki.ig  the  machines  more  compack." 

Operator  D.  "I  think  it  would  be  easay  if  the  handheld  device  were 
placed  on  a stand  for  when  you  are  sitting  down." 

Operator  E.  "The  handle  on  this  mode  should  be  made  bigger." 

Operator  F.  "Make  it  smaller." 


Checksheet 
Operator  A.  (None! 


Operator  B.  "The  good  thing  about  this  mode  is  that  the  checkers 
won't  have  any  expensive  piece  of  equipment  with  them  and  it 
wouldn't  get  in  their  way  or  get  damaged.  AI30,  everything  is  Just 
copied  off  the  label  which  makes  it  easier.  All  the  paperwork  can 
be  kept  on  one  clipboard." 

Operator  C.  {None} 

Operator  D.  {Hone} 


Operator  E.  {Hone} 

Operator  F.  "I  hate  it,  too  much  paper  work." 


Because  of  the  way  in  which  thi3  question  was  worded,  and 
because  it  followed  immediately  the  question  asking  for  equipment 
rating,  most  of  the  additional  comments  elicited  had  to  do  with 
matters  of  hardware  design.  Some  operators  offered  no  final 
comments,  as  noted,  which  suggests  that  they  felt  their  evaluation 
had  already  been  covered  adequately  In  previous  questions. 
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M,!DE  COMPARISON 


The  exact,  questionnaire  format  used  for  the  direct  comparison  of 
ail  three  data  entry  mode3  is  illustrated  in  Appendix  F to  this 
report.  The  questionnaire  sequence  will  be  followed  here  in 
reporting  the  operators'  comparative  evaluation  of  modes. 

Performance  Comparison 

The  first  question  asked  the  operators  for  ratings  of  ten 
different  aspects  of  task  performance,  on  a scale  from  0 to  IOC, 
comparing  tne  three  data  entry  modes.  Mean  ratings  are  presented  in 
Table  5-1,  which  follows  closely  the  questionnaire  format. 

Table  5-1 

Mean  Ratings  for  Different  Aspects  of  Task  Performance 
in  Comparing  Data  Entry  Modes 


Performance  Aspect 

On-Line 

Terminal 

Digital 

Recorder 

Checksheet 

Entering  the  TCN  designator 

93 

72 

76 

Entering  general  shipment  data 

95 

78 

78 

Reviewing  pre-stored  shipment  data 

96 

68 

81 

Entering  specific  piece  data 

93 

83 

78 

Detecting  errors  in  pre-stored  data 

7** 

67 

70 

Correcting  errors  made  during  data  entry 

93 

88 

82 

Ease  of  learning  to  use  input  mode 

95 

87 

90 

Interest  in  data  entry  job 

95 

84 

85 

Performance  level  achieved  in  one  day's  use 

90 

88 

88 

Physical  equipment  used  with  this  input  mode 

98 

73 

69 

Overall  Mean 

92 

79 

80 

As  one  might  expect,  these  comparative  ratings  are  quite  similar 
to  those  made  by  the  operators  when  evaluating  the  data  entry  modes 
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separately . A product  moment  correlation  between  these  30  mean 
r' tings  and  the  derived  mean  values  of  corresponding  ratings  made  in 
tne  separate  node  evaluation  is  statistically  significant  (r=.65, 
p<.0C5),  indicating  consistency  in  operator  judgement.  Again  a 
preference  for  the  on-line  terminal  is  apparent,  although  all  three 
data  entry  modes  are  rated  well. 

practical  Usefulness 

The  next  question  asked  the  operators  to  assess  the  practical 
usefulness  of  the  three  data  entry  modes: 

Considering  your  overall  experience,  do  you  believe  any  of  the 
three  input,  modes  tested  here  could  be  used  effectively  in  the 
actual  work  situation  at  the  truck  docks?  If  YES,  which  modes 
do  you  think  could  he  used? 

All  six  operators  indicated  that  the  on-line  terminal  could  be 
uaea  on  the  truck  dock.  Only  two  operators  considered  the  digital 
recorder  acceptable  for  use  on  the  job.  Three  of  the  operators 
judged  the  cheeksheet  an  acceptable  data  entry  technique.  This 
pattern  of  response  is  clearly  consistent  with  the  performance 
ratings  reported  above,  confirming  a preference  for  the  on-line 
; terminal. 

Operator  Characteristics 

The  rext  question  asked  the  operators  their  opinion  as  to  what 
kind  of  man  would  be  best  suited  to  the  data  entry  job: 

Suppose  that  one  of  your  unloading  crew  must  be  responsible  for 
entering  shipment  data  in  the  actual  work  situation,  using 
whatever  input  mode  is  available.  How  would  you  choose  which 
man  should  be  responsible  for  this  data  entry  Job?  What 
characteristics  should  he  have  to  handle  the  job  well? 

Operator  A.  "More  than  likely  rank  or  say  one  has  a light  duty 
excuse  or  is  driving  a fork  lift.  It  reall  doesn't  matter  any  one 
could  handle  the  Job.  Some  one  who  is  not  childish  and  would  use 
the  in3turment  as  a tool  and  not  a toy." 

Operator  B.  "Mostly  a person  who  would  be  conscious  of  what  he  is 
doing.  Someone  who  could  do  it  fairly  easy  and  with  as  few  mistakes 
as  possible." 

Operator  C.  "He  should  oe  trained  on  how  the  input  mode  operates. 

I would  choose  the  man  that  is  interested  the  most." 
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operator  I'.  "Trie  mar.  who  check  *3  tne  truck  shoul  1 be  the  person  who 
uses  tne  input  aod«; . Each  code  were  not  tc  difficult  to  learn  so 
any  person  should  be  able  to  use  them." 

Operator  t.  "He  roust  first  be  responsible  enough  to  handle  the 
machine  and  willing  to  check  and  recheck  all  data  that  he  has  and 
put  in  that  wnicn  he  does  not  have." 

Operator  F.  "he  would  have  tc  be  awake" 

The  purpose  of  thi3  question  was  to  elicit  some  reflection  of 
tne  operators'  view  of  tne  requirements  of  tne  data  entry  Job.  It 
seeri3  apparent  that  they  regard  the  data  handling  Job  a3  important, 
one  requiring  responsible  performance  and  careful  attention  to 
detail . 

Job  Satisfaction 

The  next  question  asked  the  operators  to  indicate  whether  thi3 
is  the  sort  of  Job  they  would  like  to  do: 

Considering  your  experience  over  several  days  of  testing,  how 
willing  would  you  be  to  handle  such  a data  entry  job  in  your 
real  work  situation? 

Interesting  job,  glad  to  do  it 5 

Tough  job,  do  it  if  I had  to 

Share  the  Job  with  others,  do  it  sometimes...  3 

Would  prefer  not  to  do  it 

Definitely  not  the  Job  for  me.... ___ 

Five  operators  indicated  that  the  data  entry  task  was 
sufficiently  interesting  that  they  would  be  willing  to  do  It  in 
their  actual  work  setting,  although  two  of  these  operators  also 
checked  the  "sharing"  alternative.  The  sixth  operator  indicated 
that  he  would  prefer  to  share  such  a job  with  others.  These 
positive  responses  probably  reflect  the  generally  high  morale 
induced  by  the  test  environment.  A more  extended  program  of  field 
testing  would  be  required  to  confirm  acceptability  of  the  data  entry 
task  as  a real  job. 
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Other  Coni  went  3 


A final  question  gave  the  operators  a chance  to  volunteer  any 
last  words  on  the  subject: 

Whatever  further  comments  you  can  make  concerning  the  data  entry 
job,  the  variruj  Input  modes,  and  the  test  procedures  will  be 
most  welcome. 

Operator  A.  (None} 

Operator  B.  "The  on-line  terminal  would  make  a lot  of  people  more 
interested  in  the  job.  It  is  much  better  tan  using  pencils  and 
paperwork,  which  becomes  boring  very  easy." 

Operator  C.  "I  think  in  my  own  opinion  that  the  input  mode  I would 
like  to  work  with  most  is  the  online  terminal." 

Operator  D.  (None) 

Operator  E.  "The  handle  on  the  digital  recorder  should  have  a 
larger  handle.  And  the  on  line  terminal  should  be  able  to  be  stored 
by  haveing  it  able  to  hook  apart  or  together  some  where  along  the 
caple  thus  preventing  unnecessary  damage." 

Operator  F.  (None) 

Then  the  operators  had  completed  this  final  questionnaire,  they 
were  asked  to  participate  in  a short  debriefing  session  where  an 
attempt  wa3  made  to  elicit  any  additional  comments  and  to  clarify  if 
necessary  their  written  evaluations.  In  particular,  each  group  of 
operators  was  asked  to  explain  their  stated  preference  for  the  on- 
line terminal  in  face  of  the  fact  that  on  each  day  of  testing  they 
had  all  seen  the  man  using  the  manual  checksheets  finish  first. 

Tneir  comments  in  response  to  this  challenge  restated  the  theme 
apparent  in  their  questionnaire  answers,  namely  that  the  checksheet 
mode  involves  too  much  paper  shuffling. 

It  may  be  inferred,  although  the  idea  was  never  made  explicit  by 
the  operators  themselves,  that  shuffling  paper  imposes  a continuous 
burden  of  attention.  That  is  to  say,  in  completing  a checksheet  the 
operator  must  decide  everything  for  himself  and  keep  careful  track 
of  what  he  is  doing.  In  using  the  on-line  terminal,  the  computer 
can  provide  much  of  this  guidance,  and  the  operator  can  simply 
accomplish  the  data  entry  sequence  as  a series  of  simple 
transactions  requiring  less  continuous  concentration  of  his 
attention. 
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SECTION  VI 


RECOMMENDATIONS 


The  overall  results  of  this  initial  test  program  look 
encouraging.  Test  operators  were  able  to  achieve  generally  good 
levels  of  performance  with  all  three  data  entry  modes,  with  no  prior 
training  in  data  handling  procedures.  The  operators  themselves  were 
unanimous  in  recommending  the  on-line  terminal  for  actual  use  in 
trick  dock  data  entry,  and  several  operators  recommended  the  digital 
recorder  and  checksheet  for  potential  use  at  the  truck  dock,  as 
documented  in  Section  V of  this  report.  On  the  basis  of  the 
positive  results  of  this  initial  testing,  it  is  appropriate  here  to 
consider  what  are  the  next  steps  to  take  toward  eventual 
implementation  of  improved  data  handling  in  the  air  transport 
system. 

Although  many  questions  can  be  explored  in  laboratory  testing  of 
tne  kind  described  in  this  report,  final  recommendations  for  system 
implementation  can  be  made  with  confidence  only  after  prototype  data 
handling  methods  have  been  tested  under  operational  conditions.  It 
is  recommended,  therefore,  that  the  on-line  terminal  undergo  field 
testing  at  a hAC  truck  dock  to  assess  its  usefulness  for  the  data 
entry  task.  A proposed  plan  for  such  field  testing  is  currently 
being  developed  and  will  be  reported  later  this  year. 

in  this  initial  test  program  the  on-line  terminal  permitted 
adequate  speed  of  data  entry  for  all  but  the  most  difficult  test 
loads.  In  field  testing  the  on-line  terminal,  the  time  required  for 
data  entry  using  this  device  could  be  decreased  in  several  ways. 
First,  the  actual  data  entry  job  should  be  analyzed  to  determine 
just  what  is  the  minimum  set  of  data  items  which  must  be  entered  in 
the  real  job,  presumably  some  subset  of  the  items  used  in  initial 
testing.  Second,  some  means  should  be  found  to  create  a machine- 
readable  tag  to  be  affixed  to  each  piece  of  cargo  unloaded  at  the 
truck  dock,  combining  the  TCN  shipment  identification  with  an 
additional  code  for  each  separate  piece  within  a shipment.  Both  of 
these  recommendations  were  discussed  earlier  in  Section  III  of  this 
report . 

Accuracy  of  data  entry  using  the  on-line  terminal  can  be 
Improved  over  the  levels  achieved  in  this  initial  test  program. 

Given  a definition  of  the  actual  data  entry  job  to  be  accomplished, 
the  computer  software  for  sequence  control  of  the  on-line  terminal 
should  be  redesigned  prior  to  field  testing,  to  incorporate  the 
specific  improvements  recommended  in  Appendix  B to  this  report. 
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Computer  processing  of  data  inputs  should  include  special  roucine3 
to  avoid  confounding  of'  0 and  0 in  keyed  entries  for  TCN,  as 
recommended  in  Section  IV.  If  any  significant  portion  of  the  actual 
job  includes  items  available  as  advance  shipment  data,  then 
provision  should  be  made  field  testing  to  permit  optional  data 
review  as  well  as  data  entry,  as  recommended  in  Sections  lit  and  IV, 
since  data  review  proved  both  fast  and  accurate  in  this  initial 
testing. 

With  regard  to  the  digital  recorder,  the  picture  is  somewhat 
different.  This  mode  of  data  entry  permitted  adequate  speed  in 
initial  testing,  but  the  absence  of  on-line  sequence  control  placed 
higher  demands  on  the  test  operators,  some  of  whom  were  not  able  to 
maintain  a high  level  of  accuracy.  A general  account  of  this 
problem  was  prov  ded  in  Section  IV  of  this  report,  and  a more 
detailed  analysis  of  factors  contributing  to  format  errors  i3 
provided  in  Appendix  E.  The  digital  recorder  cannot  be  recommended 
for  field  testing  unless  changes  are  made  in  its  keyboard  design  ano 
data  entry  procedures. 

One  possible  approach  to  improving  performance  using  the  digital 
recorder  would  be  to  redesign  its  keyboard,  following  the 
recommendations  presented  in  Appendix  C,  to  simplify  the  logic  of 
data  inputs.  If  such  a keyboard  redesign  is  undertaken,  it  should 
be  optimized  for  the  actual  data  entry  task  to  be  performed  at  the 
truck  dock.  Then  the  improved  keyboard  and  data  recording 
procedures  should  be  evaluated  in  further  laboratory  tests  to 
confirm  acceptable  accuracy  of  data  inputs  before  the  digital 
recorder  can  be  recommended  for  truck  dock  use. 

There  is  another  possible  approach  to  improving  off-line  digital 
data  recording  which  has  not  been  discussed  elsewhere  in  this 
report.  One  might  reconfigure  the  Teroiflex  keypad/display  device 
30  that  it  could  be  used  "o^ff-line"  to  store  data  inputs  on  disk 
under  control  of  a local  microcomputer.  Such  disk  records  could  be 
read  into  the  primary  data  processing  system  at  any  convenient  time 
rather  than  requiring  piece-by-piece  interaction  with  an  on-line 
terminal.  But  the  microcomputer  could  provide  sequence  control 
assistance,  with  displayed  prompts  to  the  operator  on  an  immediate 
basis  as  needed,  thus  compensating  tor  the  major  deficiency  in  the 
digital  recorder.  Use  of  the  same  Termiflex  device  for  data  input, 
either  on-line  or  off-line,  would  nelp  ensure  reliable  operator 
performance  in  his  job,  and  hence  increase  the  credibility  of  off- 
line recording  as  a potential 1 backup  to  on-line  data  entry. 

Furtner  analysis  is  needed  to  estimate  toe  comparative  cost  and 
the  expected  performance  capabilities  which  could  be  achieved  by 
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redesigning  the  digital  recorder  or  by  reconfiguring  a Termiflex  for 
off-line  use.  Depending  upon  the  results  of  that  analysis,  one  or 
the  other  approach  may  eventually  be  recommended  for  field  testing 
as  a backup  alternative  to  the  on-line  terminal,  for  potential  use 
in  work  settings  where  on-line  data  processing  is  not  available. 

With  regard  to  the  checksheet,  several  significant  questions 
remain  unanswered.  In  this  initial  test  program,  the  checksheet  was 
demonstrated  to  be  the  fastest  mode  of  data  entry  for  potential  use 
at  the  truck  dock.  But  no  measures  are  yet  available  for  the 
accuracy  of  data  entry  using  this  mode,  as  discussed  in  Section  IV 
of  this  report.  A program  of  follow-on  testing  should  be  undertaken 
to  determine  how  well  the  checksheet  records  generated  in  this 
initial  test  program  can  be  transcribed  into  digital  form,  both  in 
terms  of  the  time  required  and  the  accuracy  of  the  final  digital 
records. 

Such  follow-on  testing  will  soon  begin  at  MITRE.  Clerical 
personnel  will  work  at  on-line  display  stations  using  a keyboard  to 
enter  data  from  handwritten  checksheets.  Results  of  that  follow-on 
test  program  will  be  reported  a3  available. 

Whatever  the  results  in  terms  of  data  accuracy,  it  seems  certain 
that  both  these  methods  of  data  transcription  would  involve 
significant  delays  in  effective  data  availability  subsequent  to 
cargo  unloading  at  the  truck  dock,  as  discussed  in  Section  III  of 
tnis  report.  It  is  recommended  that  a broader  analysis  of  system 
requirements  be  undertaken  to  determine  the  impact  of  delays  in  data 
availability  resulting  from  such  transcription  processes.  Such  an 
analysis  may  eventually  disqualify  the  checksheet  as  a useful 
alternative  to  other  modes  of  data  entry  at  the  truck  dock. 

i 

Pending  the  results  of  sdch  analysis,  and  the  results  of 
follow-on  testing  of  transcription  speed  and  accuracy,  it  would  be 
premature  to  recommend  the  checksheet  for  field  testing  at  this 
time.  The  checksheet  was  the  least  preferred  mode  of  the  operators 
in  this  initial  test  program,  as  discussed  in  Section  V,  and  some 
significant  evidence  of  its  effectiveness  must  be  confirmed  in 
follow-cn  testing  before  checksheets  could  be  recommended  as  a 
backup  for  on-line  or  off-line  key  entry  in  operational  use. 
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APPENDIX  A 


KEY  ENTRY  DEVICES 


In  this  initial  test  program,  the  Termlflex  on-line  terminal  and 
the  Source  2001  digital  recorder  were  selected  3imply  on  the  basis 
of  tneir  convenient  availability  in  MITRE's  Data  Handling 
Applications  Center.  Perhaps  other  equipment  would  have  served 
equally  well  for  te3t  purposes.  The  particular  devices  tested  here 
should  each  be  considered  as  representative  of  a broader  class  of 
commercially  available  equipment.  Neither  device  was  especially 
designed  for  MAC  air  cargo  data  handling,  but  both  have  found 
successful  application  in  similar  record  keeping  tasks.  The 
following  paragraphs  provide  a more  complete  description  of  the 
capabilities  of  these  devices  than  was  presented  in  the  body  of  this 
report . The  information  was  derived  from  technical  brochures 
publisned  by  the  respective  manufacturers. 


TERMIFLEX  HT/2 

The  on-line  terminal  tested  here  is  designated  the  Terraiflex 
HT/2  by  its  manufact jrer , Termiflex  Corporation,  17  Airport  Road, 
Nashua,  New  Hampshire  03060.  The  HT/2  is  connected  to  the  DHAC  NOVA 
computer  via  an  early  version  of  the  manufacturer's  PS/1A  power 
supply.  The  HT/2  is  also  designed  for  use  in  conjunction  with  an 
acoustic  coupler,  power  supply,  and  carrying  case,  designated  the 
TC/1  Terraicoupler . 

Dimensions  of  the  handheld  terminal  are  approximately  5 x 11  x 
Id  cm.  Dimensions  of  the  power  supply  are  8 x 12  x 24  cm.  The 
terminal  weighs  approximately  0.7  kg  (1,5  lb).  The  power  supply 
weighs  2.7  kg. 

Power  requirement  for  the  HT/2  terminal  is  15  watts  (+5V  # 2A, 
-5V  § .3A,  +12V  § . 1A , -12V  § . 1A , and  -9V  # .2A).  The  PS/1A  power 
supply  operates  with  105  to  130VAC,  50/60  Hz  § 0.5A.  It  is 
Interfaced  directly  to  the  NOVA  with  an  RS232C  interface,  usually 
operated  at  120  characters  per  second. 

Communication  speed  from  the  terminal  is  selectable  at  10,  15, 

30  or  120  characters  per  second.  Selectable  parameters  Include  line 
adjust;  oad,  even,  mark  or  space  parity;  half  or  full  duplex 
transmission;  and  choice  of  either  upper  or  lower  case  characters 
for  normal  transmission. 
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Tne  HT/2  provides  a two-line  display  of  10  characters  each, 
permitting  full  alphanumerics  with  both  upper  and  lower  case. 
Character  size  is  about  5 x 7 mm  formed  by  a 5 x 7 LED  dot  matrix 
refreshed  more  than  8C  lines  per  second.  The  display  includes  an 
incoming  data  indication,  and  alternative  cursor  symbols  indicating 
next  character  position,  case,  control  mode,  special  3hift 
operation,  and  clear  to  send. 

The  HT/2  keyboard,  illustrated  in  Figure  3.  includes  a 20-key 
pad  on  the  face,  plus  three  special  shift  keys  on  its  right  side, 
permitting  generation  of  all  128  ASCII  characters  plu3  break.  The 
keyboard  features  multi-key  lockout,  and  lock/unlock  for  case.  As 
used  in  tnis  initial  testing,  some  of  the  special  control  keys, 
shown  in  the  left  column  and  bottom  row  of  Figure  3.  were 
interpreted  by  on-line  computer  software  as  signalling  various 
different  special  functions  required  in  the  data  entry  sequence,  as 
described  more  fully  in  Appendix  B to  this  report. 

The  HT/2  features  a control  switch  for  line/off/local  operation, 
an  audible  "pell”  code,  and  audible  indication  of  improper  keyboard 
operation.  The  HT/2  also  includes  a 1000-cnaract.er  memory  which 
permits  scrolling  to  view  previous  displays,  controlled  by  a 
thumbwheel  on  the  left  side  of  the  terminal.  This  scroll  memory 
capability  was  not  used  in  initial  testing,  and  probably  would  not 
be  needed  for  actual  data  entry  at  the  MAC  truck  docks. 

Current  Termiflex  equipment  specifications  state  that  it  will 
operate  over  a temperature  range  of  0*  to  50*C  (32*  to  120*F),  under 
relative  humidities  ranging  from  5 to  95  percent.  This  equipment,  is 
listed  under  Government  Services  Administration  Contract  GS-00C- 
00510,  with  current  price  of  tne  HT/2  set  at  $1570,  the  TC/1  priced 
at  $58u,  and  the  PS/ 1 A at  $160  for  single  unit  purchases. 


SOURCE  2001 

The  digital  recorder  tested  here  is  designated  the  Source  2001 
Portable  Data  Terminal  by  it3  manufacturer,  MSI  Data  Corporation, 
1381  Fischer  Avenue,  Costa  Mesa,  California  52627.  To  transmit 
recorded  data  by  telephone,  the  Source  2001  is  plugged  into  another 
device,  the  Source  2300  Communication  Module,  which  permits  acoustic 
coupling  to  a telephone  handset.  Transmission  speed  is  1200  bits 
per  second,  but  allows  effectively  1*0  characters  per  second  to  be 
transmitted. 

Dimensions  of  tne  Source  2001  recorder  are  approximately  9 x 18 
x 26  cm.  The  recorder  weighs  3-6  kg  (8  lb)  including  a self- 
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contained  battery  pack  and  an  optional  strip  printer.  Power  is 
supplied  by  a rechargeable  12.5V  nickel  cadmium  battery,  or  the 
recorder  can  be  operated  from  a standard  110V  AC  power  source. 

The  recorder  is  designed  using  solid-state  integrated  circuits, 
witn  erase  and  read/write  recording  heads,  and  single-track,  digital 
self-timing  recording.  Data  are  recorded  in  ASCII  code  on  a 
removable  tape  cassette  (Phillips  type).  Cassette  capacity  is 
50,000  characters  per  side,  which  far  exceeds  what  would  be  required 
even  for  a large  truck  load  of  cargo.  Playback  speed  is  15 
characters  per  second. 

Functional  controls  for  the  recorder  include  record,  rewind, 
play,  and  off.  Operator  and  equipment  errors  are  signalled  by  a red 
light,  an  audible  tone,  and  keyboard  lockout. 

The  optional  strip  printer  use3  36  meter  (120  ft)  rolls  of  8 mm 
pressure-sensitive  paper  tape.  A dot  matrix  print  method  is  used, 
producing  a character  height  of  about  6 mm.  Functional  controls  for 
the  printer  include  on/off,  forward/reverse  scan,  and  step.  An 
alternative  option  for  the  Source  2001,  not  available  on  the  device 
tested  here,  is  an  LED  display. 

Tne  keyboard  usually  provided  with  the  Source  2001  contains  keys 
only  for  digits  0-9,  ♦,  -,  =,  period,  clear,  ID,  and  two  user- 
specified  keys.  In  this  initial  test  program,  it  should  be  noted 
that  the  digital  recorder  was  provided  with  a special  keyboard, 
illustrated  in  Figure  5,  designed  and  built  at  MITRE  to  provide  a 
full  alphanumeric  capability.  As  it  turned  out,  this  MITRE  keyboard 
proved  deficient  in  several  respects.  Improvements  to  its  design, 
to  optimize  its  use  for  this  particular  data  entry  application,  are 
recommended  in  Appendix  C to  this  report. 

The  Source  2001  digital  recarder  is  no  longer  available  as  a 
production  model,  although  reconditioned  units  can  be  obtained.  It 
has  been  replaced  by  an  improved  model  designated  the  Source  2100, 
which  offers  a slightly  faster  transmission  rate  and  a somewhat 
larger  optional  display  (12  characters  rather  than  10).  Technical 
details  of  tne  Source  2100  can  be  obtained  from  its  manufacturer, 

MSI  Data  Corporation. 
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APPENDIX  b 


SEQUENCE  CONTROL  POP  THE  ON-LINE  TERMINAL. 


To  illustrate  the  nature  of  the  interactive  sequence  controlling 
use  of  the  on-line  terminal,  the  following  pages  tabulate  the  basic 
displays  which  appeared  at  different  stages  of  tne  data  entry 
process.  Tnis  tabulation  provides  an  arbitrary  reference  number  for 
each  display,  assigned  for  convenience  in  this  discussion,  along 
with  a facsimile  of  tne  display  itself,  plus  occasional  annotation. 

The  display  window  of  this  on-line  terminal  is  small,  just  two 
rows  of  10  symbols  each.  The  interactive  sequence  wa3  designed  to 
maximize  use  of  tnat  limited  display,  by  breaking  the  entire  data 
entry  task  into  simple,  discrete  transactions,  and  by  compressing 
displayed  messages  required  for  operator  guidance  into  as  few  word3 
as  possible. 

Tne  displays  used  fell  roughly  into  three  categories  - 
questions,  prompts,  and  transient  advisories  - and  are  shown  thus  in 
this  tabulation.  Questions  offered  the  operator  a choice  in  the 
interactive  sequence  (e.g.,  whether  to  review  previously  reviewed 
shipment  data,  display  1 20),  or  asked  him  to  confirm  the  correctness 
of  previously  entered  data.  Prompts  signalled  the  operator  that  he 
could  key  new  data  onto  the  display,  or  correct  previously  entered 
data.  Transient  advisories  appeared  as  needed  to  alert  the  operator 
to  detected  errors  in  entered  data,  or  to  remind  him  of  the  next 
step  in  the  data  entry  sequence. 

For  convenience  of  discussion,  the  total  data  entry  job  can  be 
considered  as  accomplishing  several  different  tasks.  The  first-  task 
involves  load  identification,  illustrated  in  displays  010-059. 

These  displays  were  included  only  for  purposes  of  demonstration,  to 
snow  how  this  task  might  be  accomplished  in  actual  truck  dock 
operations.  These  displays  were  used  by  the  observer  to  initiate  a 
test  session,  but  were  not  used  by  the  operators. 

The  second  general  La3k  involved  shipment  identification, 
illustrated  in  displays  112-129,  which  re;  cesented  the  first  actual 
task  of  data  entry  during  testing.  An  operator  began  his  job  by 
entering  a TCN  for  the  first  piece  of  cargo  to  identify  its 
shipment,  in  display  112.  Alternatively,  he  could  enter  a two-digit 
index  code,  as  explained  earlier  in  the  body  of  this  report. 

Various  computer  checks  were  applied  to  his  input,  as  indicated  by 
displays  113-115.  If  a new  TCN  was  entered,  the  computer  would 
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assign  it  an  index,  in  display  117,  which  the  operator  could  note 
for  future  use  if  he  wished  to  do  so. 


(A  deficiency  in  the  control  program  is  apparent  at  this  point. 
There  snould  be  greater  feedback  linking  TCN's  to  indices,  and  vice 
versa.  Imagine  that  the  operator  makes  a mistake  in  reading  his 
index  list,  and  keys  the  wrong  index.  U3ing  the  present  program  he 
might  not  notice  his  error,  and  so  proceed  to  enter  the  right  data 
for  the  wrong  shipment.  It  would  help  if  for  whichever  shioment 
designator  the  ocerato"  entered,  either  TCN  or  index,  the  co.  outer 
would  respond  with  a display  of  the  other  designator,  to  give  him  a 
chance  to  check  it.  In  that  way,  perhaps  the  convenience  of  index 
codes  could  be  preserved  without  their  pitfalls.) 

In  this  program,  sequence  control  branches  depending  on  what 
shipment  has  been  designated.  For  the  first  piece  of  a shipment 
with  advance  data,  the  operator  is  required  as  his  next  task  to 
review  all  shipment  data,  in  display  118  followed  by  display  210  (or 
211  if  HDD  is  blank).  For  the  first  piece  of  a "new"  shipment  the 
operator  is  required  to  enter  such  data,  in  display  119  followed  by 
display  211.  In  either  case,  for  subsequent  pieces  in  the  same 
shipment  the  operator  is  given  a choice  whether  or  not  to  review 
general  shipment  data,  in  display  120.  Generally  he  would  choose 
not  to  review  shipment  data  again  (the  review  option  was  selected  8 
percent  of  the  time,  in  only  46  instances  out  of  579  choices),  and 
so  would  proceed  directly  to  enter  piece-specific  data,  in  display 
129  followed  by  display  312. 

The  task  of  reviewing  or  entering  general  shipment  data  was 
mediated  by  the  sequence  beginning  with  display  210  or  211.  If  a 
data  item  already  had  a value,  whether  pre-stored  from  advance 
shipment  data  or  entered  previously  on-line,  the  sequence  control 
presented  that  value  for  review,  in  displays  210,  220,  230,  etc.  If 
the  operator  confirmed  the  displayed  value  by  keying  YES,  the  review 
sequence  continued  to  the  next  item.  If  he  keyed  NO,  then  the 
sequence  shifted  to  the  matching  data  entry  display  - 212,  222,  232, 
etc. 

Various  computer  checks  were  applied  to  each  data  entry.  If  the 
entry  seemed  improper,  a transient  error  message  was  shown,  as 
illustrated  in  displays  213,  223,  233,  etc.,  after  which  the  same 
data  entry  display  wa3  shown  again.  The  error  messages  were 
designed  to  be  as  informative  as  possible,  within  their  brief 
limitations.  A scanning  of  the  error  messages  listed  here  will 
indicate  the  kinds  of  data  checks  which  were  included  in  this 
initial  test  program.  More  specific  checks  could  be  designed  for 
actual  use  in  truck  dock  data  entry. 
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When  a data  entry  passed  the  computer  checks  correctly,  program 
control  moved  on  to  the  next  item  in  the  sequence,  showing  either 
the  next  review  display  or  the  next  data  entry  display  as 
appropriate.  If  the  operator  himself  decided  that  a previous  entry 
might  be  wrong,  he  could  use  the  general  BACKUP  option  to  move 
backward  through  the  sequence,  step  by  3tep,  seeing  either  the 
review  display  for  each  item  or  a data  entry  display  for  blank 
items. 

Review/entry  of  general  shipment  data  was  followed  by  entry  of 
piece-specific  data,  in  display  289  followed  by  di-.Dlay  310  or  312. 
Entry  of  PIECE  number  illustrates  that  potential  data  (.hocks  are  not 
limited  to  comparison  against  fixed  criteria,  but  can  also  be  made 
contingent  on  other  data  entries.  In  this  example,  if  the  PIECE 
entry  exceeds  the  value  already  entered  for  TOTAL,  the  operator  is 
required  to  review  TOTAL,  in  displays  314  and  315,  followed  by 
display  280.  A broader  example  of  this  sort  is  provided  by  the 
error  check  for  TAG  entries  which  will  not  permit  the  operator  to 
duplicate  any  TAG  already  assigned  to  another  piece  of  cargo. 

The  review  display  for  TAG  (display  340)  was  unusual  in  that  the 
computer  was  programmed  to  anticipate  data  entries,  on  the 
assumption  that  the  real  operator  would  assign  tag  identifiers  in 
sequence  tc  pieces  as  they  are  unloaded.  The  only  other  example  of 
tnis  kind  was  the  review  display  for  POE  ("TO"),  display  250. 

Tne  tabulated  display  sequence  indicates  a program  flaw  at  the 
conclusion  of  tre  entry  of  piece-specific  data.  Since  it  was 
assumed  that  the  disposition  of  individual  pieces  of  cargo  could  not 
be  known  in  advance,  r.o  review  display  for  DISP  was  provided.  If  an 
operator  had  ju3t  entered  a DISP  and  tnen  realized  it  was  wrong,  he 
could  not  BACKUP  to  correct  it.  Future  versions  of  this  program 
should  provide  a display  350  to  permit  review  of  DISP  entries,  and 
following  DISP  entry  some  display  from  which  BACKUP  is  possible. 

When  the  operator  completed  the  entry  of  piece-specific  data, 
program  centre*  cycled  back  to  the  task  of  identifying  the  shipment 
for  the  next  piece  of  cargo,  display  359  followed  by  display  112. 
This  cycling  continued  throughout  a test  session  until  data  for  all 
pieces  had  been  entered,  at  which  point  the  operator  would  signal 
STOP. 


The  STOP  input  initiated  a final  bookkeeping  sequence, 
illustrated  in  displays  400  through  442,  whicn  was  used  by  the 
observer  to  terminate  the  test  session.  The  test  operator,  his  Job 
done,  simply  handed  the  terminal  to  his  observer  who  dealt  with 
these  final  chores. 
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Tnroughout  the  control  program,  the  interactive  sequence  was 
designed,  insofar  as  possible,  to  be  foolproof  against  mistaken 
operator  inputs.  That  is  to  say,  the  operator  could  enter  wrong 
dajta,  within  the  limits  of  tne  programed  data  checKS,  but  he  could 
not  enter  wrong  sequence  commands  which  might  "confuse"  tne  control 
program.  At  branching  nodes  of  tne  program,  represented  by  the 
question  displays,  tne  operator  could  input  only  YES  or  NO.  Any 
ct.ner  input  was  rejected,  with  the  exception  of  implicit  optior.3 
sjch  as  BACKUP.  (Trie  actual  incidence  of  wrong  replies  to  YES/NO 
questions  was  less  than  one  percent,  only  41  instances  out  of  a 
total  of  49 1 responses,  generally  representing  an  impulsive  attempt 
to  correct  a wrong  data  item  directly  rather  than  first  signalling 
that  it  was  wrong.  ) 

This  rejection  of  random  inputs  was  programmed  with  a nicety 
which  seems  in  retrospect  to  have  been  wasted  effort.  Every 
question  was  programmed  to  have  an  aiternative  display  format,  so 
that  whenever  the  operator  responded  inappropriately  a different 
version  of  the  same  display  would  appear.  For  example,  display  230 
might  first  read  "PRIJhITY=2?",  but  after  a wrong  response  would 
reappear  in  altered  form  as  "PHIJhITY=2  YES/ NO?"  (Because  of  their 
redundancy,  such  alternate  display  forms,  display  231  in  this 
example,  have  not  been  included  in  the  tabulation  of  displays 
presented  here. ) 

Tne  purpose  of  tnis  particular  design  tactic  was  to  ensure  that 
for  every  operator  action  (input)  there  would  be  some  visible 
reaction  (changed  display  output)  from  the  computer,  an  elementary 
design  rule  fo  interactive  sequences  of  this  sort.  It  probably 
would  suffice  just  as  well,  however,  simply  to  program  each  question 
display  so  tnat  it  re appears  accompanied  by  an  auditory  signal 
whenever  a wrong  input  is  made.  That  simpler  expedient  is 
recommended  for  future  versions  of  this  program. 

The  sequence  control  program  for  this  initial  testing  would 
probably  nave  to  be  revised  in  other  ways  for  actual  use  on  the 
truck  dock.  For  test  purposes,  the  task  required  entry  of  all  data 
items  from  tne  shipping  label.  In  actual  operations,  it  is  possible 
that,  some  of  tnese  items  could  be  omitted  from  the  entry  sequence, 
so  that  only  a subset  representing  the  most  critical  data  would  have 
to  oe  entered. 

In  actual  use,  3till  other  kinds  of  changes  will  prove 
desirable.  The  distinction  between  general  shipment  data  and  piece- 
specific  data  need  not  be  maintained  In  the  control  program.  Such 
transitional  guidance  displays  33  116-129  and  26 9 should  be  omitted 
as  unnecessary  to  the  operator.  PIECE  should  Pe  entered  before 
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TOTAL,  reversing  displays  282  and  312,  so  that  the  entry  sequence 
for  those  items  corresponds  to  their  order  in  the  shipping  label 
format.  The  review  display  for  PIECE,  display  3 10,  should  be 
eliminated,  since  the  occasional  appearance  of  that  display  tended 
to  confuse  the  test  operators,  interrupting  their  regular  data  input 
sequence. 

Several  changes  should  also  be  made  to  the  keypad  used  with  the 
on-line  terminal.  Labelling  the  BACKSPACE  key  simply  with  a left- 
arrow  proved  too  cryptic.  That  key  should  be  given  some  sort  of 
name,  such  as  EFiASE.  The  CANCEL  key  should  be  eliminated,  since  it 
was  seldom  used  properly.  The  on-line  sequence  to  modify  the  timing 
of  transitional  displays  should  bs  eliminated,  and  the  key  used  to 
initiate  that  sequence  should  be  modified  in  function  to  provide  a 
simple  flip-flop  between  fast  (.5  sec)  or  slow  (1.5  sec)  time 
intervals. 


Pisplay  Questions  Psta  "ntry  Transient 
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919 

9'2 

n 

922 

912 

902 

iru? 

9'2 

?C1 


RFAPT  TO 
ONtOAP?. 


pock: 


CAPFIrP: 


iTRonTs  j 


gbl: 


liET-  GBL : 


FkTFR  1-2 
PIGITS 


ENTFF  P 

STB3QIF 


FKTP  P 
STHBOLF 


Ifrtpf  rror 

i;r  nogy 


A displayed  question  sack  (?)  is 
used  throughout  to  denote  that  a 
TFS  or  KO  input  is  required. 

A displayed  colon  (:)  denotes  that 
soae  data  input  say  be  required. 

The  period  (.)  is  used  here  to 
indicate  displayed  cursor  position. 

The  asterisk  (•)  is  used  here  to 
indicate  displays  which  are 
accoapanied  by  an  auditory  signal 
("beep"),  aostly  error  aessages. 


?52  repeats  itself  until  the 
operator  signals  STOP. 


91°  is  a siaple  advisory  rather 
thar.  an  error  aessage.  The  "beep" 
is  onitted. 
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Display  Questions  Data  Pntry  Transient 
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SJliC5«0l_iiSflii£ic£tion 


EWTEB 

ITCH;. 


CHECK  1CH 
DATA?. 


• ESTEP  14 
STHBOLS 

• OB  2-DIGIT 

IHDEE 

• IlHDEX  JJK 
HOT  rODHP 

• H EH  IHDEX* 

HJL__ 

Ihost  check 
Itch  data 

Ihow  fhtes 
rrcH  cata 


NOW  EHTEB 
PC  DATA 


A feu  error  eessages  east  be  output 
as  a sequence  of  displays,  in  order 
to  e.-olain  available  options.  The 
plus  ( '1  is  used  here  to  indicate 
the  seco.'d  display  in  the  sequence. 


An  underline  is  used  here  to 
indicate  a variable  cisplay 
eleaent.  The  underline  sas  not 
actually  displayed. 

Although  11?  is  not  an  error 
aessage,  the  "beep"  is  used  to 
call  attention  to  its  appearance. 

It  uas  displayed  for  tuice  the  tiee 
interval  used  for  error  sessages. 


/ 


Display  Questions  Data  Entry  Transient 
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21? 

212 

213 

22? 

222 

223 

23? 

232 

233 
2<*? 
262 
2«3 

2 -? 
2*2 
2*3 
2*  ? 
2P2 
2t  3 
2i? 
212 
273 
2P? 
282 
2P3 

2R9 


FDD.IJI?. 


PROJECT* 

in?- 


Ipriopitt*  J 
IT. 


fhon* 

[mm2. 


POD: 


• ENTER  3 
STHBOLS 


PROJEC^: 


ENTER  3 
STHBOLS 


PRIOPI^T: 


FROH: 
j 


ENTEP  1,2, 
3 OR  9 


ENTER  f I 
STHBOLS  1 


P«£M7. 


FOD*it4?. 


CONSIGN*** 

HlllV • 


TOTH!  PCS  * 
NNNg?. 


ENTER  3 
LETTERS 


{enter  3 
(letters 


Icons  tgne*: 


• Ifnter  * 

I SYMBOL 


IOTRI  PCS:| 

I.  f 


U DI GITS 
HATIH'IH 

NON  ENTEP 
PC  DATA 


This  series  of  displays  permits 
sequential  review,  or  entry,  of 
data  itess  appearing  on  a shipping 
label.  Where  advance  data  are 
available,  or  data  have  already 
been  entered,  the  iteas  are 
displayed  in  query  fora  for 
confirmation.  Where  no  data  have 
been  entered,  the  blank  itoas  ate 
displayed  as  prompts  for  data 
entry. 

The  data  validation  checks  for  all 
of  these  iteas  are  rudimentary,  as 
can  be  seen  from  the  advisory  error 
messages.  More  complicated  checks 
could  be  programmed  if  needed. 


This  question  was  exceptional  in 
that  B2V  was  displayed  when  there 
had  been  no  previous  entry  of  POE. 
If  all  truck  cargo  arriving  at  a 
terminal  did  in  fact  have  the  sane 
POE  code,  then  this  entry  could  be 
eliminated. 


Bore  complicated  checks  could  be 
made  here,  to  ensure  that  entered 
POD  matches  a possible  destination. 


During  testing,  the  actual  total 
never  exceeded  8,  but  the  variable 
display  element  is  shown  here  as 
being  potentially  larger. 
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Display  Questions  Data  Fntty  Transient 
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Esiisii'siitii.st.EisifirssfisiliE.aaia 


This  question  was  asked  only  when  a 
TOTAL  o'  1 had  been  entered.  It 
prowed  .onfusing  to  the  test 
operators  and  should  be  eliainated 
in  future  versions  of  the  prograa. 


319 

*12 

313 


PIECF«1?. 


iPIFC*: 


U DIGITS 
HAXIflUH 


3 ■*« 
315 
329 
32? 
323 
139 
332 
3 3 

3 UjJ 

3U? 

3U3 


|w*ir,HT» 


CUBr* 
HNN  N ? . 


ItaO'BnnnnnI 

I?.  1 


• 2222  looks 
TOO~LAf GF 

♦ BUST  CH rCK 
TOTAL 


Branch  to  2B9. 


weight: 


• Is  DIGITS 

Ihaxihuf 


;COBF: 


• -»  DIGITS 
■ATT1UF 


[digits 


Advance  data  on  weight  and  cube 
could  be  reviewed  only  for  single 
piece  shipments. 


In  this  question  the  displayed  TAG 
was  1 higher  than  the  last  TAG 
entered,  assuming  that  the  operator 
assigned  tag  nuabers  in  sequence. 


3ut» 
352 
3r  3 

3*» 


• 2*2222  has 

!If*”uSFD 


• 5 ST130LS 
11  A X I B UB 


EHTTF  STOP 
IF  POUF 


A flaw  in  the  prograa:  since  no 
disposition  data  were  pre-stored, 
a review  question  was  oaitted,  and 
it  was  not  possible  to  BACKUP  to 
correct  a wrong  entry  for  DISP. 

Branch  to  112. 
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Display  Questions  Data  Sntry  Transient 
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>w 

ONLOAD 

DONS?. 

»92 

NOT  I FT 
OBSEPTEP 

aia 

LIST 

LOAD?. 

029 

PRTNT 

FILP?. 

a ->9 

ISTAPT  H*TT 
h.nm?. 

i 

#32 

jlOADs 

unp 

[SIGNOFP?. 

4h2 

ISIGNED  OFfI 

La?  NN:NN 


This  advisory,  not  transient,  ms 
included  for  test  purposes.  It  was 
accoapanied  by  3 "beeps",  which 
alerted  the  observer  to  sake  the 
reaaining  entries  needed  to 
coaplete  the  test  session. 


. / 
>/ 
/ ' 
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APPENDIX  C 


IMPROVED  PROCEDURES  FOR  DIGITAL  RECORDING 


In  the  body  of  this  report,  several  problems  were  noted  with 
regard  to  the  format  control  logic  for  data  entry,  and  the 
procedures  for  error  correction,  associated  with  use  of  the  digital 
recorder  in  this  initial  test  program.  A more  detailed  analysis  of 
errors  of  entry  format,  presented  in  Appendix  E,  suggests  that  some 
of  these  problems  are  probably  associated  not  with  the  general 
concept  of  digital  recording  as  a data  entry  mode,  but  rather  with 
deficiencies  in  the  particular  implementation  chosen  for  testing. 
That  is  to  say,  some  of  these  problems  could  be  effectively 


eliminated  by  an  improved  keyboard 
procedures  for  data  entry: 

Observed  Problem 

Operators  may  forget  to  push  TCN 
key  before  entering  TCN  or  index. 

Operators  may  confuse  data  items 
grouped  in  a "line". 

Operators  may  forget  tc  key 
ENTER. 

Operators  may  hit  indicator  keys 
by  mistake  when  keying  data. 


Operators  may  get  confused  using 
BACKSPACE  key. 


Operators  may  misuse  CANCEL  key. 


Operators  may  forget  to  key 
indicator  twice  to  signal  line 
re-entry . 

Operators  may  forget  to  key  TCN 
twice  to  re-enter  TCN. 


design  permitting  simpler 


Possible  Solution 

Make  the  TCN  key  more  prominen.. 
in  size  and  position. 

Provide  a separate  key  to 
indicate  each  data  item. 

Omit  ENTER  if  individual  field 
indicators  are  used. 

Provide  an  extra  margin  of 
space  between  indicator  keys  and 
data  keys. 

BACKSPACE  would  be  used  less 
often  if  re-entry  of  individual 
data  items  were  easy. 

CANCEL  would  not  be  needed  if 
data  items  could  be  re-entered 
easily. 

Item  re-entry  need  not  require 
double  keying  of  indicator. 


A special  key  is  needed  for 
unambiguous  correction  of  a 
wrongly  input  TCN. 
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Observed  Problem 


Possible  Solution 


Operators  may  forget  to  assign  Index  assignment  should  be  an 

an  index  when  beginning  data  entry  optional,  discrete  action  at  the 
for  a new  multi-piece  shipment.  end  of  each  data  entry  session. 

A keyboard  layout  incorporating  these  suggested  improvements  is 
diagrammed  in  Figure  C-l.  Here  keys  are  arranged  in  relation  to 
their  intended  function,  with  indicator  keys  and  other  special  keys 
around  the  margins  of  the  keyboard.  Data  entry  for  each  piece  of 
cargo  would  begin  with  the  large  TCN/INDEX  key  at  the  top  left.  The 
indicator  keys  for  items  of  general  shipment  data  are  arrayed  across 
the  top  and  down  the  left  side  of  the  keyboard,  corresponding 
approximately  to  the  format  of  those  items  on  a shipping  label. 
Similarly,  the  indicator  keys  for  specific  piece  data  are  at  the 
bottom  of  this  keyboard,  just  as  those  da'.j  items  are  displayed  at 
the  bottom  of  a shipping  label.  Special  keys  required  for  error 
correction  are  grouped  at  the  top  right  of  this  proposed  keytoard. 

Although  not  indicated  in  the  diagram,  the  various  special  keys 
could  be  color  coded  according  to  their  function.  Indicator  keys 
for  TCN,  TAG  and  DISP,  representing  the  minimum  data  items  to  be 
entered  for  any  piece,  might  be  red.  Indicator  keys  for  piece- 
specific  data,  i.e.,  PIECE,  TOTAL,  WT,  CUBE,  might  be  orange,  since 
these  represent  items  which  must  usually  be  entered.  Indicator  keys 
for  general  shipment  data  items  might  be  yellow.  Keys  used  for 
error  corrections  might  be  blue.  At  the  least,  the  large  TCN  key 
should  be  made  a different  color  from  the  other  keys  in  its 
vicinity,  in  order  to  add  perceptual  salience. 

It  should  be  noted  that  this  proposed  keyboard  is  designed  to 
facilitate  the  data  entry  task  as  it  was  performed  in  this  initial 
te3t  program,  i.e.,  entry  of  a complete  set  of  shipping  label  data, 
beginning  with  TCN.  In  actual  truck  dock  data  entry,  it  is  possible 
that  the  task  could  be  simplified  to  include  only  portions  of  the 
shipment  data,  and  it  is  possible  that  the  entry  sequence  would 
begin  with  the  TAG  identifier  for  each  piece  of  cargo.  If  the  task 
were  changed  in  these  ways,  the  proposed  keyboard  design  should  be 
modified  accordingly. 


APPEKDIX  D 
SAMPLE  CHECKSHEEI3 

The  following  pages  illustrate  one  complete  set  of  checksheets 
used  in  this  initial  test  program.  The  checksheets  in  this  sample 
set  provide  a high  level  of  advance  data  for  a test  load  of  low 
shipment-to-piece  ratio.  Printouts  of  advance  data  are  included  for 
9 shipments  of  tne  total  of  12  shipments  in  this  load.  The  set  of 
checksheets  begins  with  a cover  sheet  indexing  these  9 shipments 
arranged  "alphabetically"  by  TCN.  Succeeding  pages,  each  showing  a 
prominent  index  number,  display  the  advance  data  for  each  shipment. 

The  items  comprising  the  general  data  for  each  shipment  are 
arranged  in  a format  corresponding  to  the  actual  layout  of  shipping 
labels,  to  facilitate  comparison;  missing  items  are  denoted  by  an 
empty  field  of  dots.  Lines  for  piece-specific  data  are  displayed 
below  the  outlined  label  format,  with  the  number  of  these  lines 
corresponding  to  the  number  of  pieces  known  to  be  in  the  shipment. 

In  the  checksheet  mode  of  data  entry,  the  test  operators  were 
instructed  to  review  advance  data  Items  for  accuracy  and 
completeness,  and  to  write  in  any  necessary  changes.  For  purposes 
of  testing,  the  information  actually  shown  on  a shipping  label  wa3 
assumed  to  be  correct,  and  advance  data  items  were  to  be  changed  as 
necessary  to  match  the  laf<sl.  In  the  sample  checksheets  shown  here, 
it  can  be  seen  that  correction  to  advance  data  wcs  made  for  the 
shipment  indexed  3 (wrong  consignor  code),  and  an  addition  made  for 
shipment  9 (missing  project  code).  

In  using  the  checksheet  mode  of  data  entry,  the  te^t  operators 
had  to  write  in  the  necessary  piece-specific  data  for  all  shipments. 
For  shipments  with  no  advance  data  available,  the  general  data  also 
had  to  be  written  onto  the  checksheet,  copied  from  the  shipping 
label.  Blank  checksheets  were  included  for  that  purpose,  and  three 
of  these  are  shown  at  the  end  of  the  sample  set  illustrated  hers. 

Test  operators  were  instrucuted  to  write  legibly,  and  reminded 
frequently  during  the  test  series  that  their  checksheets  would  be 
used  later  by  other  people  in  transcribing  the  written  data  to 
digital  form.  Examination  of  the  resulting  checksheets,  however, 
raises  some  question  as  to  how  accurately  such  data  transcription 
could  be  performed.  The  sample  sheets  illustrated  here  represen* 
reasonably  legible  writing  in  comparison  with  some  others  produced 
in  this  Initial  test  program. 


BI/PC-NOT  riUQ 
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fR2CICIN3  FACE 


The  general  layout  of  the  checksheets  seemed  satisfactory  in 
testing,  and  could  probably  be  recommended  as  a desirable  format  for 
practical  use.  The  layout  of  the  cover  sheet  Index  list  could  be 
improved.  In  retrospect  it  seems  a mistake  to  have  listed  TCN's 
alphabetically.  The  most  distinctive  feature  of  a TCN  tends  to  be 
its  last  group  of  four  symbols.  The  preceding  symbol  groups, 
representing  codes  for  initiating  agency  and  date,  tend  to  be 
similar,  sometimes  identical,  from  one  shipment  to  another.  During 
the  data  entry  task,  test  operators  were  observed  scanning  the  last 
column  of  symbol  groups  on  the  index  list,  looking  for  a match  with 
the  next  TCN  to  be  entered.  Such  scanning  would  be  faster  if  the 
list  of  TCN's  were  rearranged  so  that  these  last  groups  of  symbols 
were  in  numeric  order.  Such  a rearrangement  would  also  reduce  the 
possibility  of  error  in  identifying  the  proper  TCN  in  an  occasional 
instance  where  the  last  group  of  symbols  in  one  TCN  is  the  same  as 
that  in  another,  since  these  two  TCN's  would  be  displayed  together 
in  the  index  listing. 


132 


INOEX  TCN  INDEX  TCN 


*50762  3054  4043  22 

*50775  4676  5126  23 

764407  4150  0776  24 

705620  5036  0316  25 

750711  2526  6332  26 
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•2603*  5072  X030  20 

V04720  5067  0431  30 

*>-r  ‘II  nr\p  / 

* V f f ^ v •>''<-  32 

*n(^  33 

34 

35 

36 

37 

38 
30 


1 

2 

2 


40 

41 

42 


1 

t 

490762  3194  4043 

• 00 

• M * 

1 

2 1 

1 

oov 

1 

1 

1 

1 

1 • 

OSN 

PUCE  OF 
1 2 

2 2 


bEIGMT  CUBE  » T*C  OIIP 

.3.  ! tfjx ?! 

.X.  i ■•f.'.Qy/ 


I l 

I 1 I 
I I 

I I 


I I 

I 1 I 
I I 

I,,,.,  I 


134 


\ 


1 *91773  4676 

9126 

6M 

166 

i 

Ml  ' 

1 M6NH6 

2 1 
1 

1 00V 

1 

l 

l Q6M 

1 

1 

1 4T4177 

1 

Piter  op 

WEIGHT 

CUBE 

1 

T 46 

0167 

l 4 

.UP. 

.%. 

1 

1 

0SAj 

2 * 

.no. 

1 

i 

Vffff 

AW 

6 4 

.at 

1 

I 

A.W 

4 4 

.UP. 

.X. 

J 

1 

0(4407  4|)l  >770  ••• 

Z»JC  '/  * * 

040444 

OOv 

oov 

4(4407 


9 


itcc 


04 

»C!6HT 

CUBE 

» 

T*6 

01(0 

4 

m/m 

..i. 

) 

) 

fos/etf 

M.'f/ 

4 

/ ^ 

4«r'M 

/ 

• • • • 

t 

» 

iP.tU 

4 

• •/fa 

/ 

• • f • 

t 

» 

V.U& 

c Q'V/ 

• • • • • 

4 

l 

t 

A ON! 

i i 

• 9 i 

I I 

I,, ...I 


i I 

I 9 I 

I I 

l,„Ml 


136 


\ 


i rMiu  3i8i  ast« 
i emit 


atccc  of 


MCI6HT  CUBE 

.?xs  ..<? 

.firy  .a 

.?£'/  .ii. 

or  a us. 

?.}.H  l.%. 


Stt'.'fi 

9S»*s 

ViiW 

rmii 


1 


* 

■ 

i 

t rsiMi  tsai  1333 

m 

1 

• • • • 

i rsrtsi 

t i 

1 oov 

1 

1 TO  J 
1 

1 

1 

i 

i 

i 

i 

1 

PXCCI  OF  mIIOhT 

cuac  i r*6 

Oil* 

1 3 

! i>w:/ 

Z4.S 

* * 

..Vi  1 *&{?? 

.Tspco 

. 

3 3 ,4C. 

.s.  ! 

.fr/fte 

\ i 

i 

t 

x 

1 1 

1 1 

A 

1' 

1 0 1 

1 9 t 

\ 

1 1 

1 1 

1 1 

138 

■■■)' 

9 - ~~*'yrz  , y?*; * " ~ " ' . ” ' 

_ ----- 

. K v_- 

' -V,  - 

r;  ./  ■ ‘H-i-  •:  ' v 

' /*  *■ . • ; *.* 
••  1’ 

•••  V 

i 

1 FJi;t3  $76* 
1 

1 f$A*%1 

t 

1 00V 

1 

i m r 
i 

1 rsaeaa 
1 

797$ 

184 

i 

» • * » 

* 

a t 

i 

i 

t 

i 

8X«Xf  07 

HCI6HT 

CUBE 

» T*6 

’ 

_ 

oxaa 

1 • 

.m 

,'S. 

! vxp.w 

✓taaa 

t • 

m 

JS. 

! ??#?** 

ssf  ! 

3 • 

.‘AS. 

/s. 

! 

xey  ! 

4 • 

>&% 

/x 

i WJt* 

8 • 

./X 

« a 

.M 

.'X 

7 a 

m 

m 

I s 

.m 

ts: 

i ??.V.'.4 

f.rrr 

fxece 

OF 

WEIGHT 

cune 

1 TAS 

DIM 

t 

« 

.to 

! *&’.>.? 

<*r>i  >o 

4 9999 

2 

a 

.to 

1 ^syat 

$PJ?o 

3 

a 

.to. 

• J.  • 

j ?*S//2 

0 o i ">0 

4 

a 

.to 

. J.  • 

! i&'.U 

&Q!?£> 

8 

a 

.to. 

! mv.'v 

A f>.?o 

• 

6 

to. 

! 

1 • Ml  M ^ 

V04720  9067  0491  ... 

•N3960  g 

00V 

pik 

V04  720 


PIECE  OP 
1 2 

2 2 


HEIGHT  CUBE 

..‘/A  .S. 


tag  dup 

> (V.wt 
Miu 


i i 

• b i 

i t 


i i 

I 0 I 
I I 

I. ....I 


142 


/ 


Z9.f 

CAU 

to* 

AT.ttn 


i i 

i i 

• /b  * 


or 

HEXSHT 

CUBE 

t T AG 

ox*r 

X 

3.7. 

.(.. 

! 

.K. 

XX 

/... 

! 

O 

X. 

ry 

Ja?*o 

X. 

XX 

1 ?&/#> 

Mte*  o 

X. 

?.x. 

1 

$&'<:■' o 

X. 

! 3&W 

Star?  e 

X. 

3.1. 

« 

4 M • M • 

.r/.v  v 

#3, 

| ?^//s r 

* *•  1 1 n 

ms / 

• • # • 

• • • • 

i 

i 

* 


143 


~7 


Wf.trUt 

P.QO 

mff  * 


4t>L 

J 


iccc  or 


or 

KET6MT 

CUBE 

l 

T*S 

oxir 

.3. 

.7S. 

IS. 

l 

t 

WAft* 

MW  1 

.?S. 

) 

t 

?9vr*</ 

CiP.HL  1 

J. . 

• • • • 

t 

l 

l 

??st:o 

'Vdtl[.( 

• • • • 

«••• 

l 

\JL\ 


144 


/ 


HSW  w:j 
VAv.v' 1 


« 

I 


'ICC! 

or 

weight 

CUBE 

1 

T AC 

oxir 

• Ml 

/.fo 

/ / 
MM 

1 

t 

1 

Af# 

MM 

MM 

1 

1 

• • t t 

t 

1 

• • • t 

• • • • 

» 

l 

MM 

MM 

l 

1 

• Ml 

• • • • 

» 

j 

• • • • 

MM 

» 

j 

MM 

MM 

l 

i 

t • • • 

• • • • 

• 

1 

i 

It 

• • • • 

MM 

i 

l I 

I I 


APPENDIX  E 


FORMAT  ERRORS  IN  USING  THE  DIGITAL  RECORDER 


As  observed  in  the  body  of  this  report  (Section  IV),  use  of  a 
digital  recorder  for  data  entry  can  result  in  a variety  of  format 
errors  of  different  kinds.  Some  format  errors  can  be  detected  as 
logical  discrepancies  between  a data  item  actually  entered  and  ti.e 
formal  requirements  for  that  item  - a TCN  with  too  few  symbols,  or 
perhaps  an  undefined  entr  of  "6"  for  PRIORITY.  Such  errors  of 
improper  data  format  can  occur  in  any  mode  of  data  entry.  Other 
format  errors  result  from  mistakes  in  the  sequencing  of  data  inputs, 
as  for  example  a failure  to  indicate  which  data  item  is  being 
entered  next.  Such  errors  of  entry  format  are  easily  made  when 
using  a digital  recorder,  and  difficult  to  correct  in  subsequent 
analysis  of  digital  records.  That  problem  is  the  subject  of  this 
Appendix. 

Digital  records  generated  during  this  initial  test  program  were 
scrutinized  to  determine  what  kind  of  format  errors  occurred. 
Altogether  some  30  specific  types  of  error  were  identified  in  ten 
general  categories.  Some  kinds  of  error  had  to  do  with  data  format 
and  some  with  entry  format,  as  those  terms  were  used  above.  Errors 
in  entry  format  reflected  failures  to  remember  or  use  properly  the 
indicator  keys;  trouble  with  the  concept  of  line  entry,  along  with 
use  of  the  ENTER  key  to  separate  data  items  within  a line;  and 
operator  confusion  about  error  correction  procedures.  Some  kinds  of 
error  were  serious  in  their  consequences.  Other  errors  made  no 
difference  in  the  final  record  but  merely  involved  unnecessary 
effort  for  the  operator.  Some  of  the  more  serious  errors  could  be 
detected  and  corrected  to  some  degree  in  computer  processing  of 
digital  records.  Others  could  not. 


A general  categorization  of  error  types  is  presented  below, 
along  with  comments  on  the  consequences  of  each  type  of  error,  and 
what  means  might  be  taken  to  prevent  it  if  possible,  or  to  detect 
and  correct  it  otherwise. 

Errors  of  Data  Format 

Two  categories  of  error  have  to  do  with  faulty  format  of  the 
data  entered.  Errors  of  data  format  would  be  detected  and  corrected 
immediately  using  an  on-line  terminal,  but  cannot  be  corrected  in 
the  record  produced  by  the  digital  recorder. 
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1.  Missing  data . The  operators  sometimes  forgot  to  enter  a 
data  item  which  was  required.  The  consequences  of  this  error  can 
vary  considerably.  If  the  operator  fails  to  enter  an  item  of 
shipment  or  piece  data,  then  it  may  be  only  that  item  which  i3  lost 
from  the  final  data  record.  If  he  forgets  to  enter  a TCN  or  INDEX 
to  identify  the  shipment,  however,  then  the  consequences  are  more 
serious.  Data  items  for  the  new  piece  would  be  overlaid  as  supposed 
corrections  to  corresponding  data  items  for  the  last  previous  piece, 
creating  wrong  data  for  the  previous  record,  losing  the  new  record 
altogether,  and  possibly  losing  other  records  as  well  depending  upon 
how  carefully  the  operator  enters  data  for  multi-piece  shipments. 
These  errors  cannot  be  corrected  by  computer  analysis.  The  operator 
must  be  trained  to  enter  all  necessary  data,  although  it  is  clear 
that  even  a well-trained  operator  will  occasionally  become 
distracted  and  forget  an  item.  The  key  designating  input  of  the 
TCN,  initiating  the  data  entry  sequence,  should  be  made  prominent  in 
some  way,  to  try  to  ensure  that  the  operator  will  not  forget  that 
one  especially  critical  item. 

2.  Wrong  data.  The  operators  sometimes  entered  data  items 
wrongly,  perhaps  entering  a symbol  string  that  was  too  long  or  too 
short  or  recognizably  deficient  in  matching  some  other  formally 
defined  requirement.  The  consequences  of  this  error  again  vary 
depending  upon  whether  just  a single  data  item  is  faulty,  or  whether 
a wrong  TCN  has  invalidated  the  entire  data  record  for  a piece  of 
cargo.  These  errors  can  be  detected  in  subsequent  computer  analysis 
of  the  data  record,  but  cannot  be  corrected  when  an  off-line  device 
such  as  the  digital  recorder  has  been  used  for  data  entry.  The  only 
solution  to  this  problem  is  to  try  to  prevent  such  errors  by 
selecting  responsible  operators  and  training  them  in  the  importance 
of  careful  data  entry.  Even  then,  some  persistent  level  of  errors 
must  be  expected. 

Misuse  of  Indicator  Keys 

The  remaining  categories  of  error  have  to  do  with  errors  in 
entry  format.  Such  format  errors  would  be  impossible  for  the  user 
of  an  on-line  terminal.  For  the  digital  recorder,  errors  of  entry 
format  would  vary  with  the  keyboard  design  and  the  associated  logic 
of  the  data  entry  sequence. 

3.  Missing  Indicators.  The  operators  sometimes  forgot  to  hit 
an  indicator  key  to  identify  the  next  data  items  to  be  entered.  In 
this  initial  test  program,  indicator  keys  were  used  to  signal  a 
"line"  of  data  items.  The  consequence  of  omitting  an  indicator  was 
to  string  the  new  data  items  at  the  end  of  the  previous  line;  but  in 
subsequent  processing  that  previous  line  was  truncated  to  its  proper 


148 


lengt n,  so  that  new  items  were  lost.  When  the  indicator  for  a TCN 
enwry  is  forgotten,  the  situation  is  more  serious,  as  mentioned  in 
the  body  of  tnis  report.  If  the  new  TCN  is  not  recognized  as  such 
in  subsequent  computer  processing,  then  all  new  data  entries  will  be 
accepted  as  supposed  corrections  to  the  corresponding  data  items  in 
the  record  for  the  previously  entered  piece.  In  that  process,  the 
previous  record  becomes  completely  wrong,  and  the  new  record  is 
completely  lost.  Several  preventive  measures  appear  possible.  One 
approach  would  be  to  revise  the  keyboard  so  that  every  data  entry 
must  be  preceded  by  its  own  indicator,  as  recommended  in  Appendix  C. 
For  the  operator,  such  a keying  logic  would  strengthen  the  habit  of 
hitting  an  indicator  key  before  every  data  entry.  In  subsequent 
computer  analysis  of  data  records,  the  consequence  of  a forgotten 
indicator  would  be  loss  of  just  one  data  item  rather  than  a "line" 
of  items.  (The  line  concept  should  be  abandonned  in  any  case 
because  it  was  confusing  to  the  operators,  as  noted  below.)  The 
indicator  key  for  TCN  entry  should  be  made  especially  prominent,  to 
reduce  the  likelihood  that  it  will  be  forgotten.  Since  the 
consequences  of  its  omission  are  so  severe,  the  computer  software 
used  to  analyze  recorded  data  should  include  some  sort  of 
preliminary  screening  routine  to  detect  omission  of  TCN  indicators 
and  insert  them  as  needed.  In  this  initial  test  program,  a routine 
was  devised  to  scan  the  data  record  immediately  following  entry  of 
disposition  code  (properly,  the  last  item  entered  for  each  piece)  to 
determine  whether  a long  unidentified  string  of  symbols  appeared,  in 
which  case  that  string  was  assumed  to  represent  a new  TCN  and  an 
indicator  was  inserted  before  it.  This  routine  corrected  many 
omissions  properly,  but  in  a few  instances  caused  errors  of  its  own. 
Its  net  effect  was  clearly  beneficial.  A similar  screening  routine 
was  devised  to  insert  indicators  before  the  line  of  data  items 
containing  TAG  and  DISP. 

4.  Wrong  indicators.  The  operators  sometimes  hit  an  indicator 
key  accidentally,  or  hit  a wrong  indicator  key.  If  the  operator 
does  not  notice  this  error,  some  data  loss  may  occur  by  premature 
truncation  of  line  entries.  If  the  TCN  indicator  has  been  keyed  by 
mistake,  this  can  cause  more  extensive  loss  from  the  data  record 
jeing  prepared:  the  next  data  item  entered  will  be  considered  a3  a 

new  TCN  and  subsequent  data  items  in  the  record  will  be  associated 
with  that  spurious  TCN.  Computer  analysis  software  may  detect  such 
errors  but  cannot  correct  them.  A keyboard  expanded  to  eliminate 
the  need  for  line  entry  would  reduce  the  consequences  of  wrong 
Indicator  keying.  The  TCN  indicator  key  in  particular  should  be 
positioned  so  as  to  minimize  the  likelihood  of  its  accidental  use. 

It  is  Important  also  that  if  the  operator  himself  notices  wrong 
indicator  keying  he  be  given  the  means  to  correct  his  error . In 
this  initial  test  program,  the  operators  sometimes  tried  to  use 
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BACKSPACE  or  CANCL  keys  for  that  purpose.  As  it  happens,  the 
original  data  analysis  software  did  not  anticipate  this  problem  and 
did  not  permit  such  erasure  of  indicator  entries.  The  software  was 
subsequently  revised  to  deal  appropriately  with  operator  corrections 
of  that  sort. 

Errors  in  Line  Entry 

The  concept  of  entering  groups  of  data  items  in  a single  "line", 
along  with  the  consequent  need  to  use  the  ENTER  key  to  separate 
items  within  a line,  proved  a source  of  some  confusion  to  the 
operators  in  this  initial  test  program.  The  line  concept  and  its 
techniques  of  implementation  require  the  operators  to  remember  a 
number  of  things:  to  use  appropriate  line  indicators,  to  enter 

items  within  a line  in  a fixed  order,  marking  them  with  the  ENTER 
key,  to  re-key  an  entire  line  of  items  when  correcting  one  of  them, 
etc.  When  memory  fails,  as  it  sometimes  doe3,  the  operator  may  make 
errors  in  the  format  of  line  data  entries.  Errors  of  thi3  kind 
would  not  occur  using  a keyboard  on  which  each  individual  data  entry 
could  be  signalled  using  a separate  indicator  key,  following  a 
keyboard  design  such  as  that  recommended  in  Appendix  C.  Using  such 
an  improved  keyboard,  data  items  could  be  entered  in  any  order, 
blank  items  could  be  omitted,  wrong  items  corrected  individually, 
with  no  confusion  in  concept  or  data  entry  technique  caused  by  item 
grouping  into  lines.  With  the  line  entry  logic  actually  used  in 
this  initial  test  program,  however,  a number  of  format  errors  were 
observed. 


5.  Displaced  data.  One  operator  on  several  occasions  keyed  a 
data  item  in  the  wrong  line,  at  the  beginning,  with  the  result  that 
this  item  wa3  lost  and  all  other  items  in  that  line  were  displaced 
and  recorded  wrongly.  There  is  no  reasonable  means  of  correcting  an 
error  of  this  kind  in  subsequent  computer  processing.  These  errors 
apparently  resulted  from  simple  lapses  of  attention,  since  that 
operator  usually  entered  the  same  data  items  correctly. 

6.  Blank  items.  Several  operators  when  correcting  a data  item 
at  the  end  of  a line  (e.g.,  CONSIGNEE)  sometimes  entered  blanks  for 
the  preceding  items,  rather  than  re-entering  all  items  in  the  line 
a3  they  had  been  instructed  to  do.  In  straightforward  computer 
processing  of  the  data  record,  this  lapse  in  entry  format  would 
result  in  loss  of  the  blanked  items.  The  data  analysis  software  was 
modified  so  that  blank  entries  could  not  erase  pre-stored  items  of 
advance  shipment  data,  which  represents  a solution  of  sorts.  On  the 
other  hand,  operators  sometimes  forgot  to  make  even  blank  entries 
for  preceding  items,  which  resulted  in  displaced  data.  A better 
solution  would  be  to  abandon  line  entry  altogether.  A3  a somewhat 
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different  indication  of  the  potential  confusion  caused  by  blank 
items  in  a line  entry  format,  one  operator  actually  entered  a line 
of  blanks  (denoting  no  RDD  and  no  PROJECT)  on  several  occasions,  an 
unnecessary  data  entry  which  had  no  effect  on  the  final  data  record 
but  simply  represented  wasted  effort. 

7.  hissing  ENTER *s . Operators  sometimes  forgot  to  key  ENTER 
when  needed  to  signal  separation  of  data  items  in  the  middle  of  a 
line,  which  resulted  in  loss  of  the  next  item,  possibly  a wrong 
entry  for  the  preceding  item,  and  probably  wrong  entries  for  any 
following  items.  There  is  no  feasible  correction  routine  which  can 
be  applied  in  subsequent  data  analysis.  Sometimes  the  ENTER  key  is 
not  forgotten,  but  some  other  key  is  hit  accidentally  in  its  stead. 
The  results  are  the  same  unless  the  key  hit  accidentally  is  an 
indicator  key,  in  which  case  the  confusion  is  compounded  as  in  the 
type  4 e^ror  described  earlier.  Sometimes  the  ENTER  kty  has  been 
used  properly  but  then  is  accidentally  erased  from  the  record  by 
subsequent  use  of  BACKSPACE  in  an  attempt  to  correct  the  next 
following  data  entry.  The  resulting  data  loss  is  the  same.  One 
operator  frequently  forgot  to  key  ENTER  at  the  conclusion  of  a line 
entry.  Although  these  omissions  violated  his  instructions,  it 
happens  that  they  proved  harmless  in  that  they  had  no  effect  on 
subsequent  processing  of  his  data  record. 

8.  Wrong  ENTER  *s . Operators  sometimes  accidentally  hit  the 
ENTER  key  when  they  should  not.  If  ENTER  is  struck  too  soon  when 
keying  a TCN , for  example,  that  entry  would  be  truncated  in 
subsequent  processing  of  the  data  record,  so  that  shipment 
identification  would  be  recorded  wrongly  for  that  piece  of  cargo, 
and  possibly  for  other  subsequent  pieces  as  well  in  the  case  where 
an  index  was  assigned  to  a multi-piece  shipment  as  part  of  the  data 
entry  sequence.  If  the  extra  ENTER  is  internal  to  some  other  data 
item,  then  that  item  will  be  truncated  and  stored  wrongly,  and 
subsequent  data  items  may  be  displaced  to  produce  wrong  records 
using  the  line  entry  logic.  Hitting  an  ENTER  key  twice  by  mistake 
will  also  have  the  effect  of  displacing  subsequent  data  items  in  a 
line  to  produce  wrong  records.  If  an  extra  ENTER  begins  a line  of 
data  entries,  then  all  data  items  in  that  line  would  be  displaced  in 
the  record  in  a straightforward  data  analysis.  In  this  instance, 
however,  a special  routine  can  be  devised  to  repair  the  error, 
namely  to  pre-screen  data  records  and  remove  initial  blank  entries 
from  a ±ine.  In  this  initial  test  program,  such  a routine  was 
included  in  screening  data  records,  to  remove  extra  ENTER's  at  the 
beginning  of  data  lines  (except  for  the  RDD  line),  which  corrected 
most  errors  of  this  kind.  If  the  extra  ENTER's  are  at  the  end  of  a 
line,  which  occasionally  happened,  then  no  harm  is  done  in 
subsequent  analysis  of  the  data  record.  A further  problem  with  the 
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use  of  ENTEH  was  occasionally  observed:  one  operator  sometimes 

keyed  line  entries  so  quickly  that  the  ENTER  key  was  transposed  in 
position  with  the  data  item  it  was  to  mark,  thus  resulting  in  both 
an  extra  ENTER  and  a missing  ENTER  in  the  same  line.  There  is  no 
cure  for  this  sort  of  carelessness  if  the  line  concept  of  grouped 
data  entries  is  retained. 

Mistakes  in  Error  Correction 

As  described  in  the  body  of  this  report,  the  test  operators  had 
available  to  them  several  means  of  correcting  data  entry  errors  if 
those  errors  were  detected  as  they  were  made.  Error  correction 
procedures  included  use  " BACKSPACE  or  CANCL  to  erase  single 
symbols  or  data  fields  respectively,  or  double  keying  a line 
indicator  to  re-enter  a line  of  data.  All  three  of  these  procedures 
are  potentially  subject  to  misuse,  i.e.,  they  can  cause  errors  as 
well  as  correct  them.  BACKSPACE  can  be  used  to  erase  wrong  symbols, 
or  to  eliminate  wrong  indicators  or  ENTER's  (correcting  error  types 
4 and  8 above).  But  careless  use  of  BACKSPACE  can  cause  missing 
indicators  or  ENTER's  in  the  data  record  (error  types  3 and  7). 

That  seems  an  unavoidable  risk  if  a BACKSPACE  capability  is 
provided,  and  BACKSPACE  is.  a useful  capability  when  U3ed  correctly. 
The  other  modes  of  error  correction  seem  to  offer  more  hazard  than 
help. 

9.  Misuse  of  CANCL.  For  all  operators,  there  seemed  some 
degree  of  confusion  between  the  use  of  the  BACKSPACE  and  CANCL  keys. 
CANCL  was  seldom  used,  and  then  sometimes  improperly.  In  one 
instance  an  operator  was  observed  to  use  CANCL  repetitively,  under 
the  momentary  misapprehension  that  it  was  the  BACKSPACE  key,  with 
the  result  that  he  eliminated  several  preceding  items  from  the  data 
record  rather  than  several  wrong  symbols  as  he  had  intended.  It.  is 
recommended  that  a CANCL  function  not  be  included  in  any  revised 
keyboard,  except  for  a special  key  to  cancel  a wrongly  input  TCN  as 
recommended  below.  If  data  items  are  entered  separately,  each  with 
its  own  indicator,  then  a CANCL  key  would  offer  little  advantage  in 
any  case.  Corrections  to  any  individual  entry  could  be  made  easily, 
either  by  BACKSPACE  or  by  re-entry  of  the  data  item. 

10.  Misuse  of  double  indicators . In  this  initial  test  program 
the  technique  of  double  keying  indicators  to  correct  line  entries 
was  devised  for  one  purpose  only,  to  permit  unambiguous  correction 
of  a wrong  TCN  entry  in  th-*  data  record.  Double  keying  indicators 
for  lines  of  shipment  data  was  not  really  necessary,  since  the  logic 
of  subsequent  data  analysis  was  such  that  any  new  line  of  data  would 
properly  replace  the  old,  whether  a single-  or  a double-keyed 
indicator  wa3  used.  The  operators  were  not  told  this,  however, 
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since  it  was  considered  important  that  they  develop  a consistent 
habit  of  double  keying  indicators  when  making  line  corrections. 
Although  double  keying  was  not  really  needed  for  correcting  shipment 
data,  double  keying  was  essential  for  proper  correction  of  TCN*3. 
When  used  properly,  double  keying  of  the  TCN  indicator  is  a 
satisfactory  technique  for  signalling  that  a correction  must  be  made 
to  the  last  previous  TCN  in  the  data  record.  But  the  hazards  of 
misuse  are  severe.  When  trying  to  correct  a wrongly  input  TCN,  the 
operator  may  forget  to  begin  his  correction  with  a double  stroke  of 
the  TCN  indicator.  The  consequences  in  subsequent  analysis  of  the 
data  record  are  various:  the  previous  wrong  TCN  remains 

uncorrected,  with  or  without  associated  shipment  data;  and  the  new 
correct  TCN  may  be  dissociated  from  its  data  depending  upon  when  in 
the  data  entry  sequence  the  re-^ntry  of  TCN  was  attempted.  Even 
worse  is  the  situation  where  an  operator  accidentally  keys  the 
indicator  twice  when  starting  to  enter  a new  TCN:  the  new  TCN 

replaces  the  last  previous  TCN  in  the  data  record,,  the  new  shipment 
data  replaces  corresponding  items  in  the  old;  and  old  items  not 
replaced  will  persist  wrongly  in  the  new  record.  The  most  sensible 
solution  to  these  problems  is  to  abandon  the  double  keying  technique 
altogether,  and  to  include  on  the  keyboard  a new  key  used 
specifically  to  cancel  a wrongly  entered  TCN.  Such  a key  is 
incorporated  in  the  improved  keyboard  design  recommended  in  Appendix 
C.  Perhaps  that  key  can  be  given  some  notable  color  to  help  ensure 
its  proper  use  on  the  occasions  when  it  is  needed. 

Frequency  of  Format  Errors 

Having  catalogued  these  ten  general  types  of  format  errors,  the 
next  step  is  to  consider  their  frequency  and  conseauences.  A 
summary  of  the  relative  frequency  of  format  errors  of  different 
types  is  presented  in  Table  E-1.  Altogether,  1 8 3 format  errors  were 
noted.  Of  these,  24  were  harmless  in  their  consequences  and  have 
been  omitted  from  this  tabulation.  Of  the  159  errors  remaining,  76 
were  corrected  by  software  routines  used  to  screen  the  data  records 
in  preparation  for  subsequent  analysis.  These  screening  routines 
Introduced  6 new  errors,  so  that  a net  total  of  89  format  errors 
remained . 

Table  E-1  indicates  the  consequences  of  those  residual  format 
errors:  data  records  completely  lost  for  14  pieces  of  cargo, 

completely  wrong  for  21  other  pieces,  7 false  records  added,  plus 
lost  data  for  66  miscellaneous  items  in  other  piece  records,  and 
wrong  data  stored  for  65  items.  These  data  losses  would  have  beer, 
much  higher  if  no  screening  procedures  had  been  used,  as  indicated 
by  the  parenthetic  numbers  shown  in  the  table. 
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The  incidence  of  data  format  errors  (37  errors  of  typos  1 and  2 
above)  was  somewhat  lower  then  the  comparable  figure  when  these 
operators  were  using  the  on-line  termi-al  (59  errors,  as  shown  in 
Table  4-1).  This  finding  suggests  that  the  users  of  the  digital 
recorder  were  trying  to  be  careful,  since  they  could  not  rely  on  a 
computer  to  monitor  their  data  entries  and  flag  obvious  errors. 

Their  conscientious  performance,  however,  was  not  sufficient  to 
prevent  the  frequent  occurrence  of  entry  format  errors.  Some 
redesign  of  the  equipment  and  the  job  would  be  required  in  order  to 
reduce  tne  considerable  data  loss  resulting  from  errors  of  entry 
format. 

An  improved  keyboard  for  the  digital  recorder,  such  as  the 
design  recommended  in  Appendix  C,  would  simplify  the  logic  of  the 
data  entry  sequence  by  eliminating  the  need  to  enter  lines  of  data 
items  grouped  together.  By  simplifying  the  data  entry  task  in  this 
way,  it  is  probable  that  most  errors  of  entry  format  (error  types  5- 
10  above)  could  be  eliminated  completely. 

Even  with  an  improved  keyboard,  however,  some  errors  of  entry 
format  must  be  expected,  namely  the  omission  or  misuse  of  item 
indicators  (error  types  3 and  4 above).  Examining  the  frequency  of 
such  errors  as  shown  in  Table  E-1,  it  is  evident  that  errors  of 
omission  (type  3)  are  muon  more  common  than  errors  of  commission 
(type  4).  Although  keyboard  redesign  might  improve  performance 
somewhat,  it  is  probable  that  omission  of  data  indicators  would 
remain  a persistent  problem  in  a real  data  entry  job  just  as  it  was 
in  this  laboratory  situation.  If  so,  then  some  sort  of  data 
screening  routines  to  insert  missing  indicators  will  be  useful  in 
processing  real  data  records  Just  a3  they  were  needed  in  this 
initial  test  program. 

Given  an  improved  keyboard  design  and  a simpler  logic  for  data 
entry,  it  is  possible  that  the  digital  recorder  could  prove  adequate 
as  an  alternative  (or  backup)  mode  of  data  entry  for  use  instead  ojf 
an  on-line  terminal.  Further  laboratory  testing  would  be  needed  to 
assess  that  possibility.  If  an  improved  method  of  digital  recording 
could  be  developed  in  the  laboratory,  it  would  then  have  to  be  ! 
evaluated  in  field  testing  to  confirm  its  real  value.  Patterns  of 
error  might  be  significantly  different  in  real  Job  performance.  i 

\ 

In  the  laboratory  setting  of  this  initial  teat  program  the  data  \ 
entry  task  was  continuous.  As  an  operator  finished  entering  data  I 
from  one  label,  the  next  label  was  immediately  available  on  top  of 
the  remaining  pile.  Many  errors  may  have  resulted  from  haste  to 
complete  the  data  entry  task.  In  a real  data  entry  job,  the  process 
of  cargo  handling  at  truck  dock  or  pallet  pit  would  tend  to 
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interrupt  the  data  entry  task  as  the  operator  moves  from  piece  to 
piece.  The  pacing  of  data  entry  would  be  more  deliberate,  with  a 
longer  pause  between  each  piece  cf  carjo.  Under  those  conditions, 
certain  kinds  of  errors,  like  forgetting  to  key  a TCN  indicator,  may 
be  less  frequent.  Or  perhaps  they  might  be  more  frequent.  Field 
testing  will  be  needed  to  resolve  -hat  uncertainty. 

In  the  laboratory  task  used  in  this  initial  test  program,  all 
data  items  from  a shipment  label  had  to  be  entered,  including  some 
items  always  present  and  other  items  which  had  to  be  entered  only 
occasionally.  In  a real  data  entry  job,  it  might  be  possible  to 
regularize  the  task  to  require  data  entry  only  for  a minimal  subset 
of  items  always  present.  Under  those  conditions,  it  is  possible 
that  the  operator  could  develop  more  reliable  habits  of  data  entry 
and  make  fewer  format  errors  in  the  entry  sequence.  Again,  further 
testing  would  be  needed  to  assess  that  possibility. 
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APPENDIX  F 


OPERATOR  EVALUATION  QUESTIONNAIRES 


The  following  pages  illustrate  the  wo  types  of  questionnaires 
used  to  record  operatoi  evluations  of  t.ie  several  data  entry  modes. 
The  first  questionnaire,  three  pages  long,  was  completed  by  each 
operator  at  the  end  of  a day's  test  sessions  using  a particular 
mode.  It  begins  with  an  open-ended  question  intended  to  elicit  a 
general  reaction  and  comments  on  aspects  of  the  data  entry  mode 
wnich  seem  important  to  the  operator.  The  next  question  asks  the 
operator  to  rate  his  overall  evaluation,  on  a scale  which  can  later 
be  quantified  in  comparison  with  otner  ratings. 

On  page  two  of  tnis  questionnaire  the  operator  is  asked  to  rate 
performance  on  different  aspects  of  the  data  entry  task.  He  is 
asked  for  spontaneous  comments  on  his  experience  in  learning  to  use 
the  data  entry  mode,  and  then  is  asked  to  rate  ease  of  learning. 

Or.  page  three  he  is  asked  to  rate  his  interest  in  the  task, 
reflecting  motivational  factors.  He  is  a3ked  to  rate  data  handling 
performance,  a consistency  check  on  earlier  ratings.  He  is  asked  to 
rate  the  physical  equipment  U3ed.  Finally,  he  is  asked  again  for 
general  comments,  to  elicit  any  ideas  which  may  have  occurred  to  him 
while  working  on  the  questionnaire. 

The  second  questionnaire,  also  three  pages,  was  designed  for  a 
somewhat  different  purpose.  It  was  used  only  on  the  third  day,  at 
the  completion  of  all  test  sessions,  to  record  each  operator's 
evaluation  of  all  three  data  input  modes  considered  in  comparison 
with  one  another.  This  questionnaire  begins  by  asking  the  operator 
to  provide  a numerical  rating  for  each  mode  on  all  aspects  of  the 
job  previously  considered. 

On  page  two  the  operator  is  asked  to  assess  the  three  modes  in 
terms  of  their  potential  application  to  the  actual  data  entry  job. 

He  is  asked  to  describe  desirable  qualifications  for  a person  doing 
the  data  entry  job,  to  elicit  his  perceptions  of  what  talents  the 
Job  requires.  He  is  asked  to  rate  his  own  willingness  to  work  at 
such  a job,  on  a hypothetical  basis. 

On  page  three  the  operator  is  asked  for  the  last  time  to  offer 
additional  comments  or  opinions. 
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MAC  AIR  CARGO  TERMINAL  DATA  ENTRY  - INITIAL  TEST  PROGRAM 


Input  Mode: 


L.»*  Tc-fm t*>? 


D*“!-5.-.i3.1?5 Z. 


After  a day's  experience  using  this  particular  input  mode  to  enter  shipping 
data,  what  is  your  general  reaction?  Which  aspects  of  the  job  go  well, 
and  which  poorly?  What  advantages  does  this  data  input  mode  provide?  What 
disadvantages? 


"T  jLLis  4j_LL£g.  £2JXJlLl1±s:  <j>//  y^rr</  fkm 

-E  YQ  c r.zr.  .‘.r.  -j  *t  P r-tak* **  k «-  C-gr^e  Ma  u g -f  jLS./sjg 

. ^ U • c A ^ i«  <r  f Lf  cl  <"/«  » . 

Tit,  ^ k’  i *1  y fi  f.  ^ L f 4 ■ ftL'sl 

jjHixJs ±Lx 72  a...  

■gg/lcc/s jJLajj.  * caw/-/  'rt  > > • _£  rffr-t-  Lf p.  ** k r 

rygVr  4. Men* /,w/./c,r.  AS  :/■ , 


Check  one  of  the  boxes  on  the  right 
to  Indicate  your  overall  evaluation 
of  this  input  mode  for  the  task  of 
entering  shipment  data. 
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Check  to  Indicate  your  evaluation 
of  this  Input  mode  in  performing 
specific  aspects  of  the  data  entry 
Job. 

Entering  the  TCN  designator 

Entering  general  shipment  data 

Reviewing  pre-stored  shipment  data 

Entering  specific  piece  data 

Detecting  errors  in  pre-stored  data 

Correcting  errors  made  during  data  entry 


Excellent 

Good 

Adequate 

Fair/Poor 

Poor 

✓ 

■ 

■ 

□ 

,/ 

i 

■ 

/ 

M 

■ 

m 

What  are  your  general  impressions  of  your  experience  in  learning  how  to  use 
this  input  mod*.?  Which  aspects  were  difficult?  How  could  your  training 
have  been  improved? 


\ A 

\ y 


Considering  your  experience  with  this 
input  node,  rate  your  Interest  in  the 
Job  today. 


Considering  speed  and  accuracy  of  data 
processing,  and  any  other  factors  you 
think  important,  how  would  you  rate  your 
performance  using  this  input  mode? 


Excellent 

-o 

o 

o 

Adequate 

Fair/Poor 

Poor 

T 

How  would  you  ra&£  the  physical 
equipment  used  in  this  input  mode? 


What  other  comments  can  you  make  concerning  the  equipment,  or  any 
aspect  of  the  test  situation? 


w ttici 


-O- 


U K - /, 


J±fO 


J+i  /V  a,  r*  X ygsf 


0)01 


yL h ^ k r ...  dl'y- 
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MAC  AIR  CARGO  TERMINAL  DATA  ENTRY 


INITIAL  TEST  PROGRAM 


Date:  7 (T 


Now  that  you  have  tried  all  three  ,'nput  modes,  you  are  asked  to  compare 
them  In  terms  of  how  well  they  perform  t lie  data  entry  job.  In  the  table  below 
are  listed  several  aspects  of  job  performance.  Please  rate  the  three  input  modes 
on  each  of  these  aspects,  on  a scale  frem  0 to  100. 

(Imagine  that  you  are  a teacher  grading  students.  Even  your  best  student 
tna>  nOt  uesei*e  a grade  of  10C< . Gimil.iily,  you  may  decide  that  the  best 
Input  mode  deserves  a rating  no  higher  than  8C)  on  some  aspect  of  the  job.  If 
you  consider  the  second-best  input  mode  only  half  as  good,  you  would  rate  that 
at  4JD,  and  so  on.) 


Input  Modes: 


On-Line  Terminal 


Digital  Recording 
"1 


Checksheet  - 

] I 


Performance  Aspect 

Rati  nr.s 

Entering  the  TCN  designator 

to ... 

~fo^ 

to 

Entering  general  shipment  data 

mam 

ro 

to 

Reviewing  pre-stored  shipment  data 

foO  . 

to 

zo 

Entering  specific  piece  data 

'io 

to 

Detecting  errors  in  pre-stored  data 

VO 

m 

t 0 

Correcting  errors  made  during  data  entry 

to 

so 

to 

Ease  of  learning  to  use  input  mode 

msm 

to 

Interest  in  data  entry  job 

•So 

Vo 

VO 

Performance  level  achieved  in  one  day's  use 

yf  P 

?CJ 

Physical  equipment  used  with  this  input  mode 

/PC' 

■fSB 

..  i£.a. 
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Considering  your  overall  experience,  do  you  believe  any  of  the  three 
Input  modes  tested  here  could  be  used  effectively  in  the  actual  work  situa- 
tion at  the  truck  docks?  . 

If  YES,  which  modes  do  you  think  could  be  used?  7j( g 1 JE, 


J 


f.g/ beet'*  -1  "-y 


If  NO,  what  are  the  deficiencies  which  you  believe  would  handicap  use 
of  these  modes  in  the  actual  work  situation?  


Suppose  that  one  of  your  unloading  crew  must  be  responslbla  for  entering 
ship-lent  data  in  the  actual  work  situation,  using  whatever  input  mode  is 
available.  How  would  you  choose  which  man  should  be  responsible  for  this 
data  entry  job?  What  characteristics  should  he  have  to  handle  the  job  we*l? 

— U.C. "S  knu  L I t o ■ »-  r . * A/ac itCLUl iLkjC !.  rjfjiti — 

-g^-g  rai.ri.  Z1 luaiaIJ  a.koe\c^  4-Ag  /<•** • l-*T  4 Ll.- 


Considering  your  experience  over  several  days  of  testing,  how  willing 
would  you  be  to  handle  such  a data  entry  job  in  your  real  work  situation? 

Interesting  job,  glad  to  do  it \r 

Tough  job,  do  it  if  I had  to......... _____ 

Share  the  job  with  others,  do  it  sometimes... 

Would  prefer  not  to  do  it _____ 

Definitely  not  the  job  for  me 


Whatever  further  comments  you  can  make  concerning  the  data  entry  job, 
the  various  input  modes,  and  the  test  procedures  will  be  most  welcome. 


JLm h i xr * 

+U  u4-  rtnJ  C 
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If  we  are  able  to  send  you  a summary  of  test  results  some  time  during 
thenext  several  months,  what  address  should  we  use  to  reach  you? 

El  g.  WLCJT /?.  & Ki..i)  < (. 

Mt>t. 


And  finally,  thanks  for  your  help  in  this  test  program. 


